LMGQS: A Large-scale Dataset for Query-focused Summarization
Authors: Ruochen Xu
Song Wang
Yang Liu
Shuohang Wang
Yichong Xu
Dan Iter
Pengcheng He
Chenguang Zhu
Michael Zeng
Venue: EMNLP 2023 Findings (2023)
Keywords: query-focused summarization
large-scale dataset
zero-shot summarization
large language model
TL;DR: We propose a large-scale query-focused summarization dataset consisting of over one million samples.

Abstract: Query-focused summarization (QFS) aims to extract or generate a summary of an input document that directly answers or is relevant to a given query. The lack of large-scale datasets in the form of documents, queries, and summaries has hindered model development in this area. In contrast, multiple large-scale high-quality datasets for generic summarization exist. We hypothesize that there is a hidden query for each summary sentence in a generic summarization annotation, and we utilize a large-scale pretrained language model to recover it. In this way, we convert four generic summarization benchmarks into a new QFS benchmark dataset, LMGQS, which consists of over 1 million document-query-summary samples. We thoroughly investigate the properties of our proposed dataset and establish baselines with state-of-the-art summarization models. By fine-tuning a language model on LMGQS, we achieve state-of-the-art zero-shot and supervised performance on multiple existing QFS benchmarks, demonstrating the high quality and diversity of LMGQS.

LMGQS: A Large-scale Dataset for Query-focused Summarization
Ruochen Xu, Song Wang, Yang Liu, Shuohang Wang, Yichong Xu, Dan Iter
Pengcheng He, Chenguang Zhu, Michael Zeng Microsoft Azure Al
rruxx, sonwang, yaliul0,
yonwangyylliu1 0, shuowa, yicxu, iteudant {penhe, chezhu, nzeng) @microsoft com
Abstract
For each summary, write general question about the article that answered by
Instruction
Query-focused summarization (QFS) aims to extract generate a summary of an input doc- ument that directly answers or is relevant to given query. The lack of large-scale datasets in the form of documents, queries, and sum- maries has hindered model development in this In contrast, multiple large-scale high- quality datasets for generic summarization ex- ist. We hypothesize that there is hidden query for each summary sentence in a generic sum- marization annotation, and we utilize a large- scale pretrained language model recover In this way, we convert four generic summa- rization benchmarks into a new QFS bench- mark dataset, LMGQS, which consists of over I million document-query-sunee -uerr--uuurrrrrryr samples. We thoroughly investigate the properties of proposed dataset and establish baselines with state-of-the-art summarization models. By fine-tume fine-tuming a language model on LMGQS, we achieve state-of-the-art zero-shot and super- vised performance on multiple existing QFS benchmarks, demonstrating the high quality and diversity of LMGQS.
Article:
key rebel commander and fugitive from a U.S. drug trafficking... ...
Summary:
Tomas Medina was fugitive from a U.S. drug trafficking indictment
summary answers fallowing questions the
article:
1. Who was Tomas Medina Caracas?
4.
One-shot Example
Article:
For many people, being happy at work an important pursult
Summary:
1. Naomi Simson is blogger, entrepreneur, speaker
Input Document Summary
The summary answers the following questions from the
article:
Who Naomi Simson?
What is her book about?
<text>: Generated Query
Figure 1: Example prompt for query generation. The top part is the instruction, followed by the one-shot example consisting of document, summary, and query. The query for the input document (highlighted yel- low) and summary is generated by InstructGPT.
1 Introduction
The field of generic summarization (See et al., 2017; Gehrmann et al., 2018; Liu and Lapata, 2019) has made significant progress in recent years, thanks to the development of generative deep neu- ral models (Sutskever al., 2014; Vaswani al., 2017) and the availability of large-scale training data (Nallapati et al., 2016; Narayan et al., 2018; Zhu et al., 2021). However, query-focused summa- rization (QFS) presents a significant challenge due to the lack of data. Most of the available QFS cor- pora (Dang, 2006a,b; Nema et al., 2017; Baumel et al., 2016; Zhong et al., 2021) contain only few thousand documents or less, which is insufficient for training a robust neural model.
to address the lack of a large-scale QFS dataset. Hu- man annotation for QFS typically involves gener- ating suitable queries and then writing correspond- ing summaries, which is both time-consuming and expensive. Furthermore, it may necessitate a metic- ulous definition of the query scheme based on the domain of documents (Zhong et al., 2021). We hypothesize that, for pair of document and sum- mary in a generic summarization dataset, hidden queries exist that represent the information needs associated with the summary. Therefore, effi- ciently scale up annotation, we prompt the large- scale language model InstructGPT (Ouyang et al.,
We propose a Language Model Generated Query-focusee Summarization Dataset (LMGQS)

---

do
Istare
have
Prompt
Query Type
Dataset
did
does
was
could
would
his
had
whot
when
where
whom
which
whose
why
how
other
queries
queries
CNNDDM
0.08
0.16
0.01
0.02
0.02
52.36
5.36
4.29
14.21
0.2
3.98
19:21
0.1
0.29
99.61
72.54
0.98
12.00
0.17
Wh-quarics
SAMSum
0.49
0.19
0.03
0.12
0.04
66.43
7.12
6.58
7.5
0.12
0
2.79
8.29
0.39
0.87
98.93
DialogSum
78.88
1.4
3.8
0.08
CNNIDM
40.33
43.06
1.54
5.79
4,42
0.12
0:17
0.02
0.15
0:01
0
0.01
0.08
4.32
95. 14
0.56
XSUM
26,56
42.19
12.2
8.07
0.01
0.13
0.05
9.12
90.43
Yesiso queries
SAMSom
$7.96
36,78
0.79
17.88
2.52
0.03
0.03
0.01
0
0
0.01
3.99
95.93
0.08
DualogSum
59.68
27.84
0.56
4.9
1.11
0.01
0
0
94.09
0.01
Table I: Breakdown of query types on QuerySum. The upper part of the table shows the query type percentage when using wh-queries the one-shot prompt example. The lower part shows the percentage when prompting with yes/no queries. Color of blue means higher percentage while red means lower one.
2022) with documents and summaries from four generic summarization datasets to generate the hid- den queries. This approach results in the LMGQS dataset, which contains over 1.1 million triplets of document, query, and summary, encompassing wide range of document and question types.
them, CNNi and XSUM are news sum- marization datasets, where both the documents and summaries are in formal written English. SAM- Sum and DialogSum are two recently proposed dialogue summarization datasets, whose inputs are the transcripts of multi-speaker conversations.
To investigate the utility of our proposed LMGQS, we finetune pretrained language model on it. The model accepts the concatenation of the original document and generated query as input and is trained to produce the original summary. We then compare the finetuned model with vari- ous query-focused summarization models on sev- eral existing QFS benchmarks that have overlap with LMGQS under the zero-shot setting. Empir- ical results demonstrate that the model finetuned on LMGQS achieves promising performance on both single-document and multi-document QFS benchmarks, surpassing strong baselines. Simi- larly, when utilizing LMGQS for pre-finctuning, the model achieves state-of-the -frt performance in the supervised setting.
2.1 Prompt-based Query Generation
Given a document and its corresponding summary, we take advantage of the robust few-shot capabil- itics of Instruc(GPT (Ouyang et al., 2022) to gen- erate query that encapsulates the information quired by the annotator when crafting the summary. More specifically, we construct a prompt for each document- summary pair and input it into the In- structGPT model to generate the query by complet- ing the prompt. An example prompt is illustrated in Figure 1. Since InstructGPT excels at adhering to human-readable instructions and can even gener- alize to unseen instructions (Ouyang et al., 2022), begin our prompt with a clear directive for the query generation task. Following the instruction, we incorporate one-shot example of the task into the prompt, which includes a human human-written -witten query derived from a document-summar pair. We set the number of examples to be 1 based on balance between effectiveness and efficiency: during our preliminary exploration, we noticed more failure cases for zero-shot query generation, while incor- porating additional examples in the prompt would increase both the time and cost of generation.
In summary, our contributions are three-fold: (1) We introduce a novel framework for constructing a QFS dataset by converting existing generic summa- rization datasets using language models as annota- tors. (2) We present LMGQS, a large-scale QFS benchmark, to foster future research on QFS!. (3) The model finetuned on LMGQS exhibits robust generalization capability and achieves remarkable zero-shot and supervised performance other un- seen QFS test sets.
In the one-shot example, we restrict the number of queries to be equivalent to the number of sum- maries. In other words, there exists onc-to-one correspondence between the sentences in the sum- mary and the query. This constraint is imposed by appending prefix indices and appending newline characters for each summary/query sentence, as illustrated in Figure 1.
2 Dataset Creation
We choose 4 generic datasets to build LMGQS: CNN/DailyMaiil (Nallapati et al., 2016), XSUM (Narayan et al., 2018), SAMSum (Gliwa et al., 2019), and Sum (Chen al., 2021). Among
Datasct will be released after the anonymous period
Due to the domain difference between news and

---

Lon
Len
Lon (sum)
NIP
NTP
NTP
NTP
NTP
NTP (sum, query)
Prompt Query Type
Dutaset
Count
(doc)
(query)
(sum, doc)
Iquery, doc)
(dnc, sum)
(doc, query)
(query. sum)
CNN/DM
311938
677.7
29.9
52.0
21.5
36.7
87.34
94,74
44.6
70.3
XSUM
226287
368.0
10.0
22.1
42.2
43.1
92.
96.45
45,5
73.4
Wh-queries
SAMSum
92.9
41.5
76.51
87.89
DualogSum
14460
130.7
15.1
24.2
36.6
48.0
82.7
90.97
47.0
65.9
CNNDD
311920
677.7
33.7
52.0
21.5
27.4
87.34
92.6
20.7
48.2
XSUM
226276
42.2
40.4
92.37
Yow'o queries
SAMSum
16368
92.9
17.5
22.3
41.5
42.9
76.51
81.71
19.0
34.1
DialogSom
14460
130.7
16.4
24.2
36.6
37.9
82.7
87.91
27.8
47.9
Table 2: Statistics about LMGQS. "Len" stands for length and Len(string) is the count of word tokens in the string. "NTP" stands for novel token percentage. NTP(stringl, string2) computes the percentage of tokens in string that are not present in string2.
dialogue summarization, we choose different one- shot examples for the two domains. The queries for the two example pairs were annotated by the au- thors of this paper and are attached in the appendix.
queries has strong Pearson correlation (0.95) with the length of summaries, which is expected due to our one-to-one mapping between the summary and query sentences. Second, the length of queries is consistently shorter than the summary, with wh- queries slightly shorter than yes/no queries.
2.2 Prompt Query Types
Given document and summary sentence, multi- ple valid queries can be formulated. For instance. consider the summary sentence: She has released a book to encourage people to find their passion at work. One possible query is: What is her book about? Alternatively, another valid query could be: Has she released a book? To address this variety, we utilize two sets of annotated queries: yes/no queries and wh-queries. Yes/no queries correspond questions that can be answered with simple "yes" or "no". However, in the context of QFS, the summary (i.e., the answer to the yes/no query) is never a mere "yes" or "no". For example, for a yes/no query like Is he still alive?, we expect the answer to be: He was killed in an attack on a guer- rilla encampment rather than simple no. Detailed annotated queries are presented in Table 11.
We introduce the novel token percentage: NT ( age of tokens in stringl that are absent in string2. This statistic quantifies the amount of unique in- formation contained in stringl with respect string2 First, P(doc, query) is always lower than P(doe sum), indicating that the gener- ated query always contains less information about the document than the summary. Subsequently, we observe that NTP(query, doc) is in general higher than NTP(summ doc). because queries are shorter and contain more unique question words like "what" and "did". Finally, NT P(query, sum) being considerably lower than NT (sum, query) shows that the summary contains more unique in- formation than the query. Furthermore, the query includes a subset of information present in the sum- mary. For instance, a query might inquire about a specific entity in the document, while the summary addresses the query with detailed contexts and facts extracted from the document.
defined as the percent-
The type of queries in one-shot prompt sig- nificantly influences the generated queries. We provide a breakdown of query types in Table 1. is evident that when the prompt includes only wh- queries, over 99% of the generated queries are also wh-queries, with the most frequent ones beginning with "What". The same pattern applies when the prompt contains only yes/no queries. The most common queries generated by InstructGPT typi- cally start with "do/does/did" or "is/are/was/were".
In conclusion, LMGQS encompasses documents in both written and spoken languages, covering a wide range of document/summary lengths, abstrac- tion levels, and compression ratios.
3 LMGQS for QFS
2.3 Statistics of LMGQS
In this section, we demonstrate that by finetuning pretrained language models on LMGQS, one can obtain QFS model that generalizes effectively unseen tasks and domains. In particular, we fine- tuned a BART model (Lewis et al., 2020), and the resulting model, LMGQS BART, exhibits promis-
Using the aforementioned prompting method, we collected 1.138,077 document-querr--uunmay triples covering 13 different query types. Detailed statistics of the generated LMGQS dataset are shown Table 2. First, the length of the generated

---

Dataset
Size(Train/Tess
Query Example
MultiOpEd
1954/560
is protecting the environment incompatible with capitalism's 's values? snow, weather, cold, winter, temperatures, conditions, hot, morning, expected, parts
NEWTS word
2400/600
winter temperatures, hot weather conditions, a cold morning, snow is
NEWTS phrase
2400/600
expected later
This topic is about winter temperatures as opposed to hot weather conditions, cold mornings, and weather forecasts like snow being expected later.
NEWTS sentence
2400/600
Would the election of president make the eu a more accountable institution?
Debatepedia
12000/1000
Identify computer viruses detected worldwide. Include such details how they spread, what operating systems they affect, what damage they inflict, their country of origin, and their creators wherever possible. Deseribe the state of teaching art and music public schools around the world. Indicate prohlems, progress and failures.
DUC 2006
0/200
DUC 2007
0/180
Table 3: Size and example queries of QFS datasets used in evaluation.
ing performance on various QFS datasets when directly applied to the unseen test set. Moreover, when extending the fine-tuning process with several thousand in in-domain QFS data points, the resulting supervised model surpasses other strong supervised baselines.
plicitly addresses a controversial topic, a sys- tem is expected to generate a single-sentence thesis statement that summarizes the argu- ments presented. Along with ROUGE scores evaluation metrics, the paper also proposes trained classifiers to assess the correctness and relevance of the generated summary. More specifically, a stance classifier is utilized predict whether a summary shares the same stance as the news article. For example, a sum- mary that presents an opposing argument to the article might still achieve a high ROUGE score due n-gram overlap but would receive a low stance accuracy. Similarly, relevance classifier is employed to evaluate whether the summarized perspective is pertinent to the query.
3.1 Implementation Details
We fine-tuned BART-Large (Lewis et al., 2020) on LMGQS, using maximum input length of 1024 and output length of 256. The input string con- sists of a document and query, formatted as ques- tion:vi <query> 'v wentext//v<cccumeet "An" represents a newline character. We employed 8 NVIDIA Tesla V100 GPUs for training, with a batch size of 4 per GPU and an accumulation step of 8, yielding an effective batch size of 256. The BART model was fine-tuned using learning rate of 3 10-5 for 50, 000 steps, and the learning rate scheduled by polynomial scheduler with 2000 warmup steps. We set a weight decay of 0.001 and label smoothing factor of 0. 1. For supervised finetuning, we continued to finetune the LMGQS BART model with 2000 total steps and 200 warm- up steps. The implementation from Huggingface (Wolf et al., 2020) was utilized.
where
NEWTS (Bahrainian et al., 2022) dataset is a corpus for summarizing news topics. It is based on the CNN//ailymail dataset (See al., 2017) and was annotated through online crowd-sourcing. Each source article is paired with two reference summaries, each focusing on a different theme of the source document. The dataset has 3,000 source articles (2,400 for training, and 600 for testing). In addition to standard ROUGE score, the dataset is eval- uated using LDA topic model indicating the strength of the target topic for the generated summary. We follow the implementation from Bahrainian et al. (2022) to compute the topic focus score. It is expected that summaries closer to the target topic get higher topic fo- cus scores.
3.2
Datasets
We conduct evaluation of the finetuned BART- Large model (LMGQS BART) on several existing QFS benchmark datasets.
MultiOpEd (Liu al., 2021) presents an open- domain news editorial dataset specifically de- signed to support automatic perspective dis- covery in news articles. Given query that ex-

---

Model
ROUGEI
ROUGE2
ROUGEL
BERTSCORE
Relevance. Acc.
Stance Acc.
BART
28.2
11.3
27.0
88.7
91.9
72.3
Rel
28.4
11.5
27.1
88.7
93.0
72.7
Stance
28.2
88.8
91.3
73.4
Supervised Models
Rel & Stance
29.2
11.9
27.9
88.7
94.6
74.3
CNN/DM BART
26.9
10.7
25.2
86,4
99.5
68.6
LMGQS BART
31.5
13.8
29.8
89.1
96.8
78.8
InstructGPT 002
23.8
9.3
21.9
85.7
99.5
73.2
Zero-shot Models
CNN/DM BART
17.9
5,2
16.2
85.2
94.8
62.9
LMGQS BART
25.4
10.4
23.6
87.2
99.5
77.0
Table 4: ROUGE scores and accuracies of stance and relevance on MultiOpEd dataset. All baseline results except for InstructGPT are from (Liu al., 2021)
Debatepeida (Nema et al., 2017) was built on Debatepedia an encyclopedia of pro and con arguments and quotes on critical debate top- ics. The summarics are highly abstractive and not extractive in the sense that the summary does not necessarily comprise of a sentence which simply copied or shortened from the original document.
for abstractive summarization. Unlike other ap- proaches, LaQSUM models the query as hidden binary variables to indicate whether token in the document contributes to the information sought in the summary. This model does not require QFS annotation and is trained on the CNN/DM dataset.
MARGESUM (Xu and Lapata, 2021) is state- of-the-art few-shot method for QFS that requires small QFS development set.
Document Understanding Conferences (DUC) 2006/2007 set up the task to simulate real- world complex question answering. The query in this dataset cannot be answered by simply stating name, date, quantity, etc. Given topic and set of 25 relevant documents, the task is synthesize fluent, well-organized 250-word summary of the documents that an- swers the question(s) in the topic statement.
USUM++uerr adapted from GSUM (Dou et al., 2021), which is a guided summarization system. An unsupervised query-focused extrac- tive system is used to pre-extract the top-ranked sentences for each test document as guidance. The GSUM model is trained with the CNN/DM dataset.
QuerySSum (Xu and Lapata. 2020) is an extrac- tive method that uses QA datasets as distant super- vision to train an evidence estimator for identifying segments likely to answer the query and should be included in the summary.
3.3 Baselines
We compare LMGQS BART with the following baseline models:
ProphetNet (Qi al., 2020) supervised ab- stractive summarization model that predicts the next n tokens simultaneously. The results for ProphetNet are taken from the NEWTS paper (Bahrainian et al., 2022).
CNN/DM BART is a large BART model that has been fine-tuned on the query-agnostic CNN/DailyMail dataset (See et al., 2017). This serves as a baseline model for summarization based solely the input document, without considering the query in the QFS setting.
Unsupervised extractive baselines are taken from Xu and Lapata (2022). Lead and LexRank estimate sentence-leve centrality using Markov Random Walk on graphs.
InstructGPT 002 is an InstructGPT model that can be accessed through the OpenAl API using the text-davinci-000 model. A simple template, "Sum- marize by answering the following questions:", is used to link the document with the query and gen- erate content with the temperature set to 1.0, top-p set to 0.9, and maximum length LaQSUM (Xu and Lapata, 2022) is a recent model that learns latent queries from documents
QMDSCNN (Pasunuru et al., 2021) transfers the CNN/DailyMai dataset into query-focused multi- document summarization dataset and build abstrac- tive end-to-end neural network models to obtain zero-shot results on DUC 2016 and 2017 datasets.
512.
3.4 Query Unification
Different QFS datasets have different query for- mats. For instance, Debatepedia has the query for- mat of a natural question, which is the same as
'URI. at nttps://www-nln wopptt.tttttooooo gov/ projects/duc

---

LMGQS, while the majority of queries in DUC datasets are instructions such as "Discuss condi- tions on American Indian reservations or among Native American communitiess. " and "Include the benefits and drawbacks of the reservation system.". And for NEWTS, the query is a "topic" in the topic model and described in words, phrases or sen- tence.
4 Evaluation Result
4.1 Results on Single-document QFS
The table 4 presents the ROUGE scores and ac- curacies of stance and relevance for various mod- els on the MultiOpEd dataset. It can be observed that LMGQS BART outperforms other models in both supervised and unsupervised cases, achieving the highest ROUGE scores and stance accuracies in both settings. For relevance accuracy, it also achieves the best in the zero-shot setting and the second best in the supervised setting. This demon- strates the robust performance of LMGQS BART across different settings. Interestingly, in the super- vised setting, pre-finetuning on the CNN/DaillMail dataset (CNN/DM BART) actually diminishes per- formance compared to vanilla BART without pre- finetuning. This result indicates that a general sum- marization dataset may not always be beneficial for QFS and highlights the necessity for high-quality large-scale QFS datasets like LMGQS
To use LMGQS in the zero-shot setting. it is necessary to convert the queries of diverse formats into natural questions. Without an off-the-shelf tool for this task, we propose to further utilize LMGQS for the query unification task. Specifically, we finetune BART model to generate queries with the document and summary as imput. The finetuned BART model shares the same input and output as the Instruct-GPT used in Section 2.1 to generate queries from generic summarization datasets.
We denote this finetuned model as Gd,s and the finetuned model described in Section 3.1 as Gá.-s. Given original query q and document d, we first use as a pseudo "summary" and ask Gd.s-g to produce a query of of the desired format, i.e., o Gd.s-g (d. (d, q). We then use the generated query g' as the input query in the follow-up zero-shot inference to predict summary
Similarly, Table 5 presents the ROUGE scores (R-1, R-2, and R-L) and topic scores on the NEWTS dataset for different models under two cat- egories: Supervised and Zero-shot/ shot/Transfer Learn- ing. We used "w/ (query | )" to denote the results using three different granularities for the query: words, phrases, and sentences. For instance, "ProphetNet supervised w/ topic words" refers to the result ProphetNet achieved using query of topic words. Overall, the LMGQS BART models outperform other baselines in terms of ROUGE scores, with the LMGQS BART w/ topic words model achicving the highest scores in the zero-shot setting and the LMGQS BART w/ topic phrases model obtaining the best results in the supervised setting. Additionally, the LMGQS BART w/ topic sentences model achieves the highest topic score among all models in both zero-shot and supervised settings, closely approaching the topic scores of the ground truth. Without fine-tuning on any super- vised data, LMGQS BART exhibits a significant advantage over the supervised ProphetNet models in terms of ROUGE scores and topic scores. The supervised results also reveal that LMGQS remains more beneficial even when some in-domain super- vised data (2,400 training samples from NEWTS) is accessible.
Ga.g-s (')
The query unification is used to generate queries for NEWTS, DUC 2006, and DUC 2007 dataset. We quantitatively and qualitatively verified its ef- fectiveness in section 4.3.
3.5 Mutli-documeet Query Focused Summarization
Since LMGQS contains only single-document QFS data, the fine-tuned model Gd.q-s can generate summaries based individual document-query pairs. To evaluate zero-shot multi-document QFS, we adopt straightforward iterative approach from previous works by Baumel et al. (2018); Xu and Lapata (2022). Given a cluster of documents and a query, we first rank documents using term frequency-nnven inverse document frequency, then gener- ate summary for each ranked document. The final summary is chosen from the top-rante top-ranked list. Fol- lowing the list order, we successively concatenate a summary its token overlap percentage with the selected summaries is below threshold, e.g., 50%, until the total length of chosen summarics reaches a predefined token budget (e.g., 250 tokens).
Table 6 presents the ROUGE scores on the single-document QFS dataset Dehatepedia for var- ious models, classified into unsupervised, super- vised, and zero-shot/transfer learning categories.

---

Category
Model
R-1
R-2
R-L
Topic Score
ProphetNet supervised w/ topic words
31.9
10.8
20.7
0.136
ProphetNet supervised w/ topic phrases
31.6
10.4
20.2
0.147
ProphetNet supervised w/ topic sentences
31.4
10.0
20.0
0.163
CNN/DM BART w/ topic words
34.5
11.5
22.1
0.176
Supervised
CNN/DM BART w/ topic phrases
34.3
11.3
22.0
0.178
CNN/DM BART w/ topic sentences
34.5
11.5
22.0
0.174
LMGQS BART w/ topic words
34.4
11.9
22.5
0.178
LMGQS BART w/ topic phrases
34.6
12.0
22.8
0.180
LMGQS BART w/ topic sentences
34.6
12.0
22.7
0.180
CNN/DM BART
31.2
10.4
20.8
0.125
Plug and Play Language Model
29.6
9.1
18.8
0.148
Customizable Abstractive Topic-hased
30.1
9.4
19.1
0.152
Zero-shot/Transfer Learning
Summarization
InstructGPT 002
32.7
10.7
21.3
0.184
LMGQS BART w/ topic words
33.3
11.2
21.6
0.141
LMGQS BART w/ topic phrases
32.6
10.8
21.0
0.161
LMGQS BART w/ topic sentences
32.4
10.5
20.9
0.187
Ground Truth
0.193
Table 5: ROUGE scores and topic scores NEWTS dataset. In NEWTS, the topic query is represented in three
formats: word, phrase, and sentence, All baseline results except for InstructGPT 002 are from (Bahrainian al., 2022). InstructGPT 002 takes topic words as query, which performs the best among all topic formats.
Calegory
Model
R-1
R-2
R-L
from two candidates, given the query and the con- text document. If both summaries were of equal quality or the query was unanswerable from the document, they would mark it as a "Tie." In the blind test, annotators preferred the LMGQS BART model's output 18 times, the reference 15 times, and selected "Tie" 17 times. This indicates that LMGQS has a higher quality compared existing
LEAD
15.9
Unsupervised
LexRank
15.1
DDA
7.4
7.2
Supervised
BERTAbs+Rank
19.2
10.6
17.9
BERTAbs+Coneat
26.4
11.9
25.1
BERTAbs
13.3
2.8
2.8
CNN/DM BART
21.4
18.4
InttructGPT 002
21.8
18.8
2cro-shot/haasst Leariing
GSUM+Qucry
21.2
18.2
LaQSUM
23.5
20.6
benchmark datasets like Debatepedia. Additionally,
LMGQS BART
21.0
we observe that model finetuned on LMGQS will not just summarize the document but also tends to answer the question by giving a direct answer and giving stance of supporting opposing the statement in the query.
Table ROUGE scores on single-document QFS
dataset Debatepedia. Baseline results (except for In- structGPT 002) are reported from Xu and Lapata (2022).
LMGQS BART achieves the highest ROUGE scores, surpassing all other models in the zero- shot/transfer learning category.
4.2 Results on Multi-document QFS
Table 7 presents ROUGE scores for various summa- rization models on multi-document QFS datasets. namely DUC 2006, DUC 2007. The models are cat- egorized into Upper Bound Baselines, Distantly Supervised, and Few- or Zero-shot Abstractive cat- egories. Note that MARGESUM this category is few-shot model while the others are zero-shot ones.
worth mentioning that our model distilled from InstructGPT outperforms the teacher model in the all single-document QFS datasets.
4.1.1 Human Study
A recent study by Laskar et al. (2022) discovered that some queries have no relation to the input doc- uments. To investigate this, conducted a human study comparing the LMGQS BART model's 's out- put with the Debatepedia reference. Human anno- tators were instructed to choose the better summary
Among the zero-shot abstractive models, LMGQS BART exhibits the second-best perfor- mance in terms of ROUGE scores on both DUC 2006 and DUC 2007 benchmarks, only trailing behind its teacher model, InstructGPT 002. We

---

DUC 2006
DUC 2007
Category
Model
R-1
R-SU4
R-1
R-SU4
Gold
45.4
11.2
16.8
47.5
14.0
18.9
Oracle
47.5
15.8
20.2
47.6
17.1
20.9
Upper Bound & Baselines
Lead
32.1
5.3
10.4
33.4
6.5
11.3
LexRank
34.2
6.4
11.4
35.8
7.7
12.7
QuerySum
41.6
9.5
15.3
43.3
11.6
16.8
Distantly Supervised
Bart-CAQ
38.3
7.7
12.9
40.5
9.2
14.4
PQSum
40.9
9.4
14.8
42.2
10.8
16.0
MARGESUM
40.2
9.7
15.1
42.5
12.0
16.9
QMDSCNN
31.1
6.3
10.9
34.1
7.6
12.5
CNN/DM BART
38.3
7.8
13.1
40.2
9.9
14.6
Few- Zero-shot Abstractive
GSuM+Query
38.1
7.9
13.1
39.5
9.5
14.3
LQSUM
39.1
8.5
13.7
40.4
10.2
15.0
InstructGPT 002
41.1
9.4
15.3
42.0
10.7
16.1
LMGQS BART
40.0
9.0
14.0
41.3
10.6
15.3
Table 7: ROUGE scores on multi-document QFS dataset DUC 2006, DUC2007.
means the model is few-shot instead of zero-shot. Baseline results (except for InstructGPT 002) are reported from Xu and Lapata (2022).
hypothesize that the primary reason for this is the prevalence of queries in the DUC datasets that are presented in a human-readable instruction format, which inherently favors the Instruction-folloving nature of InstructGPT. Despite being consider- ably smaller model, LMGQS BART still demon- strates promising cnttucttionfollloviin capabilities by leveraging our query unification method.
4.3 Ablation Study of Query Unification
To evaluate the efficacy of query unification, we perform ablation study compare the quality of automatically generated queries o' Gd.x-n q). For comparison, we manually create a query tem- plate to transform the query into a natural language question. The template is selected separately for the NEWTS and DUC datasets, and the authors uti- lize the generation on the development set of these datasets to carefully refine the template, as shown in table 8. In Figure 2, present comparison of ROUGE-2 scores between LMGQS BART when employing 1) manually crafted query templates or 2) automatically generated queries from query uni- fication. Other ROUGE scores are presented table 9 and 10 in appendix. It is evident that query unification holds advantage over the handcrafted template, despite the latter necessitating access to validation set and meticulous tuning from human experts.
Handcraft Query Template
Automatic Query Unrlication
12
ROUGE1
Conclusion & Future Works
NEWTS wi words
NEWIS wi topl ahravon
NEWTS topi sentencee
DUC 2006
DUC 2007
We introduce a novel large-scale dataset, LMGQS, for query-focused summarization (QFS), address- ing the scarcity of large-scale benchmarks in this domain. In order to reduce the laborious human effort required for QFS, we utilize large-scale lan- guage model generate hidden queries from ex- isting query-agnostic summarization datasets. By performing standard finetuning on LMGQS, we attain sate-of-the-are zero-shot performance across multiple QFS benchmarks. The resulting zero-shot
Figure 2: Ablation study on the effect of query unifica- tion. For simplicity, we only present ROUGE-2 score in this figure.

---

NEWTS
DUC
Replace the instructions 'describe', 'indicate", 'include', 'summarize' with 'What are the"
Template
Templute: What are [topical words, phrases, or sentences]?
Describe the activitics of Morris Docs and the Southern Poverty Law Center.
Original query
Topics words; winter temperatures, hot weather, a cold momning
What winter temperatures? What hot weather What are cold morning?
What the activities of Morris Dees and the Southern Poverly Law Center?
Templuted query
conditions?
Table 8: The manual template of queries that not in question format.
model can be further enhanced by finetuning on labeled QFS datasets, achieving the state-of-the- art supervised performance. In addition to QFS, another promising avenue for future research is question answering (QA). Considering the typically extensive context in open domain QA, query- focused summary of the context could prove advan- tageous for downstream QA models.
summary between two options in a comparison task. To minimize confirmation bias, the order of the candidates was randomized and hidden from the annotators. No payment was involved in this human study, and the authors exercised their best efforts to minimize any inadvertent biases in the annotation process.
Limitations
6 Ethics Statement
One limitation of LMGQS is that is solely in English, lacking the extensive multilingual cov- erage of other languages. Additionally, the data creation procedure is cost-inefficient due to the necessity for numerous API calls to InstructGPT. Our human study involves authors of this paper and might be subject to confirmation bias. Lastly, the generated queries are primarily divided into two categories: yes/no queries and wh-queries. A more fine-grained approach to control the types of queries based on the properties of the document and summary is currently lacking. For instance, if summary emphasizes location over time, a query beginning with "where" would be more appropriate than one starting with "when".
In this study, we acknowledge the ethical concerns related to the use of InstructGPT, a pre-trained lan- guage model that has the potential to generate toxic biased outputs, make up facts, and produce sex- ual and violent content without explicit prompting (Ouyang et al., 2022). To mitigate these risks, we employed the content filtering function of the Ope- nAI API calls 3. This filter runs both the prompt and complction through an ensemble of classifi- cation models designed to detect and prevent the output of harmful content. A small fraction of our prepared prompts were filtered by the OpenAl API of InstructGPT, and discarded the correspond- ing samples in our proposed LMGQQ dataset.
We acknowledge the possibility that the Instruct- GPT model might be discontinued by OpenAl in the future, rendering parts of our research irrepro- ducible. To address this issue, we plan to release the full results returned by the API calls, as well as the prompts for generating the hidden queries. This will enable the research community to construct higher-quality dataset with more advanced models in the future, reproduce our zero-shot and super- vised finetuning results, and further build upon our work.
References
Seyed Ali Bahrainian, Sheridan Feucht, and Carsten Fickhoff. 2022. Newts: A corpus for news topic- focused summarization. Findings of the Associa- tion for Computational Linguistics: ACL 2022, pages 493-503
Tal Baumel, Raphael Cohen, and Michael Elhadad. 2016. Topic concentration in query focused sum- marization datasets. In Thirtieth AAAI Conference Artificial Intelligence.
Regarding the human study, the annotators in- volved are the paper's two authors who possess pro- ficiency in English and are well-acquainted with the fucrs-focused summarization task. The annota- tors were tasked with choosing the better candidate
Tal Baumel, Matan Eyal, and Michael Elhadad. 2018. Query focused abstractive summarization: Incorpo- rating query relevance, multi-document coverage, and summary length constraints into seqßseq models. arXiv preprint 10X1vv:1000077000
'hups://caam. httrss//leam.micrssof... www.osifi.ccommmmmm wwwpic/ssoo..oo/eenuu/aaaaa servies/ooomaiccooeppss/omttttte
Yulong Chen, Yang Liu, Liang Chen, and Yue Zhang 2021. SialogSum: A real-life scenario dialogue sum-

---

marization dataset. In Findings of the Association for Computational Linguistics: ACL-WJCNLP 2021, pages 5062-5074, Online. Association for Computa- tional Linguisties.
Yang Liu and Mirella Lapata. 2019. Text summariza- tion with pretrained encoders. In Proceedings of the 2019 Conference on Empirical Methods in Natu- ral Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNTPP) pages 3730-3740, Hong Kong, China. Association for Computational Linguistics.
Hoa Trang Dang. 2006a. DUC 2005:
Evaluation of question-Focused summarization systems. In Pro- ceedings of the Workshop on Task-Focused Summa- rization and Question Answering. pages 48-55. Syd- ney, Australia. Association for Computational Lin- guistics.
Ramesh Nallapati, Bowen Zhou, Cicero dos Santos, Çağlar Gu"lçehre, and Bing Xiang. 2016. Abstrac- tive text summarization using sequence-to- to-seuencce RNNs and beyond. In Proceedings of The 20th SIGNLL Conference on Computational Natural Lan- guage Learning pages 280-290, Berlin, Germany, Association for Computational Linguistics.
Hoa Trang Dang. 2006b. Duc 2005: Evaluation of question-focuser summarization systems. In Pro- ceedings of the Workshop on Task- focceoo Summa- rization and Question Answering, pages 48-55.
Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018. Don't 't give me the details, just the summary! topic-aware convolutional neural networks for treme summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Lan- guage Processing, pages 1797-1807, Brussels, Bel- gium. Association for Computational Linguistics.
Zi-Yi Dou, Pengfei Liu, Hiroaki Hayashi, Zhengbao Jiang, and Graham Ncubig. 2021. GSum: A gen- eral framework for guided neural abstractive summa- rization. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Tech- nologies, pages 4830-4842, Online. Association for Computational Linguistics.
Preksha Nema, Mitesh M. Khapra, Anirban Laha, and Balaraman Ravindran. 2017. Diversity driven atten- tion model for query-based abstractive summariza- tion. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics ( (Vol- ume 1: Long Papers), pages 1063-1072, Vancouver, Canada. Association for Computational Linguistics.
Sehastian Gehrmann, Yuntian Deng, and Alexander Rush. 2018. Bottom-up abstractive summarization. In Proceedings of the 2018 Conference on Empiri- cal Methods in Natural Language Processing, pages 4098-4109, Brussels, Belgium. Association for Com- putational Linguisties.
Long Ouyang. Jeffrey Wu, Xu Jiang. Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Gray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human feedback. In Advances in Neurul Information Processing Systems.
Bogdan Gliwa, Iwnna Mochol, Maciej Biesek, and Alek- sander Wawer 2019. SAMSum corpus: human- annotated dialogue dataset for abstractive summa- rization. In Proceedings of the 2nd Workshop on New Frontiers in Summarization, pages 70-79, Hong Kong, China. Association for Computational Linguis- tics.
Md Tahmid Rahman Laskar, Enamul Hoque, and Jimmy Xiangji Huang. 2022. Domain Adaptation with Pre-trained Transformers for Query-Focusee Abstractive Text Summarization. Computational Lin- guistics, 48(2):279-322 320.
Ramakanth Pasunuru, Asli Celikyilmaz. Michel Galley, Chenyan Xiong, Yizhe Zhang. Mohit Bansal, and Jianfeng Gao. 2021. Data augmentation for abstrac- tive query-focused In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 13666-13674.
summarization.
Mike Lewis. Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, AbdeIrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-ssquence pre-training for natural language generation, translation, and com- prehension. In Proceedings of the 58th Annual Meet- ing of the Association for Computationnal Linguistics, pages 7871-7880, Online. Association for Computa- tional Linguistics.
Weizhen Qi, Yu Yan, Yeyun Gong. Dayiheng Liu, Nan Duan, Jiusheng Chen, Ruofei Zhang, and Ming Zhou. 2020. ProphetNet: Predicting future n-gram for parrencceto-SeqeencePeeee of the Association for Computational Linguistics: EMNLP 2020, pages 2401 2401-2410, Online, Association for Computational Linguistics.
In Findings
Siyi Liu, Sihao Chen, Xander Uyttendacle, and Dan Roth. 2021. MultiOpEd: A corpus of multi- perspective news editorials. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies. pages 4461, On- line. Association for Computational Linguistics.
A Abigail See, Peter J. Liu, and Christopher D. Manning, 2017. Get to the point: Summarization with pointer- generator networks. In Proceedings of the 55th An- nual Meeting of the Association for Computational Linguistics ( (Volume I: Long Papers), pages 1073- 1083, Vancouver. Canada. Association for Computa- tional Linguistics.

---

Ilya Sutskever, Oriol Vinyals, and Quoe V 2014. Se- quence to sequence learning with neural networks. In Advances in Neural Information Processing Systems, volume 27. Curran Associates, Inc.
Ashish Vaswami, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances Neural Information Pro- cessing Systems, volume 30. Curran Associates, Inc.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Remi Louf, Morgan Funtow- icz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma. Yacine Jernite, Julien Plu, Canwen Xu, Teven Seao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Trans- formers: -fate-of-the-ar natural language processing. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online. Association for Computational Linguistics.
Yumo and Mirella Lapata. 2020. Coarse-to-fine query focused multi-document summarization. In Proceedings of the 2020 Conference Empirical Methods Natural Language Processing (EMNLP), pages 3632-3645, Online. Association for Computa- tional Linguisties.
Yumo Xu and Mirella Lapata. 2021. Generating query focused summaries from query-free resources. In Proceedings of the 59th Annual Meeting of the Asso- ciation for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 6096- 6109, Online. Association for Computational Lin- guistics.
Yumo and Mirella Lapata, 2022. Document sum- marization with latent queries. Transactions of the Association for Computational Linguisties, 10:623- 638.
Ming Zhong, Da Yin, Yu, Ahmad Zaidi, Mutethia Mutuma, Rahul Jha, Ahmed Hassan Awadallah, Asli Celikyilmaz, Yang Liu, Xipeng Qiu, and Dragomir Radev. 2021. QMSum: A new benchmark for query- based multi-domaa meeting summarization. In Pro- ceedings of the 2021 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5905-5921, Online. Association for Computational Linguisties.
Chenguang Zhu, Yang Liu, Jie Mei, and Michael Zeng. 2021. MediaSum: A large-scale media interview dataset for dialogue summarization. In Proceedings of the 202/ Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, pages 5927-5934, Online, Association for Computational Linguistics.

---

A
Appendix
Model
NEWTS w/ topic words
NEWTS w/ topic phrases
NEWTS w/ topic sentences
DUC 2006
DUC 2007
Handeraft Query Template
30.13
25.84 84
27.94
40.54
41.14
w/ Automatic Query Unilication
33.27
32.64
32.42
40.02
41.32
Table 9; Effect of query unification in ROUGE-1
Model
NEWTS lopic
NEWTS w/ topic phrases
NEWTS lupic sentences
DUC
DUC
Handcraft Query Template
20.25
17.23
14.38
15,49
Automatic Query Unification
21.58
21.01
13.99
15.32
Table 10: Effect of query unification in ROUGE-L (NEWTS dataset) and ROUGE-SUL (DUC datasets).

---

Domain
Dialogue
BOGOTA, Colombia (CNN) A key nhel communder and (ugitte from a U.S. drig trafficking indictment was killed over the weekend in air goemilla encumpmeen, the Colombaa military sai Monday, Alleped cocaine tuthikker and FARC robel Tomas Molins Caracos, known populatly /'lll Negro Acacio!" was menneer of the high command of the Foerzas Armadas Revololionzzis de Colonilia and, accceding Colomhian estensive cucaine infliikking network He had U.S. Justice Deportment sinee 2002. le was charged with compinacy to impor coczine imo the United Stles manufactaring fund
Medina Cuacos Interpol phoso.
U.S. officias, helped munige grap's trao-luin the
Fimnr: 1u200190 just fallen live with this alvent calendde!
Awesome! warn one for my kidetveiluh I used to one every year as child! Loved thm! Valimm:: Yeah, remember! they filled with choolatestvvllaau they are differcnt these days! nuch
distributing cocaine year insurgency against governatent. U.S. officials alleged Medim. Caracas managed the rebel group's is sales of cocaine international drug wathhkeen, who tun smuggled isso the Unitd
Within
sophistcated" fitaaftaaaiis yeah, they can be fabric' wooden, shop bought! homemade, filled with various suffiiaEmm what de you ft:
also indicted United
along
Issid""e'lLuee
loys, Christmes deconation, creative stuft.
Document
charges compiiing to kidrap two U.S. oil workers from nighissinn Vireraall 1907 and bukiin onn of them for nine months unil 51 million
FARC commanders November
hands dlips, sicker, pencill & millers, asull puzzles.
swoesttiiioomm: WiW! Thadk beill! Xorel aurent adl one more way chikdren do something
poid. Officials sand the 's Foree incked
special thing well- little notes that! questions about chricims such as Whtt did ihe wise men bring?
Resporse Forss hackal elamenis the Calombiar Medins Cancras down at FARC camp the jungle the south the country. V'fter bombardmen. the trops occupied the camp, and they found equipuenn
shudetnllob:
sllk soking har
etckial aurez: reckons it prepozes them for Christas kinlimma: and makes about traditions being get very excited every time they get one cenn why! ()
robels far, with ilfes. pisools and four syseess Defanse Ministes
people/ulaaren children
Manuel Sentos nows confeemmm: The death of FF Negru Acadio was confrmellyy by various sources, including members of FARC isseff Medina Caracis FARC's 16th Frant Gurinia Fanblished in 1964 as militay wing of Colembian Communiia Part, FARC is Columbia's a oldest larges, most capable and best bqsipeee Markst rebel group, according the U.S. Department of Suze. mail friend. Journalist Femundo Rimos contributed this ropoct.
wuthem degaetment Viduado and
I. Jooms Medina Caracas was a figitive from U.S. drag
1. Emmi and Rab love the advent calendarha2, Lanzan fits inside calender varians items, for invanee, smull toys and Cheistmas
catheking indictment oz VBll Negro Acaciol" allegedly helped maugge extausive cocaine setwork. k.3. Justice Departnest indicted him wo. Colombian militay::
Summary
Her chikkrn
whenevrr they get calender.
as attack on guernilla escampment.
I. Who was Tamas Modi Caracra/eed. What was he indised Iarfaa, When var he int Tho did be die?
1. What are Eara and Rob's attindde towards advent calendar 7u2. What does Laures fit inside advent calendar"ad, What is the reaction
Wh-quesy
children when
calenda??
Yes: Was Tomas Medias Caracos fugitive No:
Do Emete and love the advest calendu/7u2, Lauren unencuuiastic about advent calendar Yes: Do Lauzen's children enjoy receiving the calendar?
Yestao Query
VBI Negro Acaciol help to fight against drogPa3. Yex: Wis he indicted by U.S. Justice Depatmeen."od No. Is he still alvv?
Wa-Iastmuctinn
suswered by it For each summary, write binary question about the article that can answered by
each summary, write general question about article
Instuction
Table 1l: One-shot prompt examples for news and dialogue domain.


Reviews:
[{"review_id": "T1Lx3OdhqB", "reviewer": "Reviewer_HLfb", "summary": "", "questions": "-", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer’s feedback! We will follow the review’s suggestions on the typo and presentation issues. \n\nTo address the **missing references of QMDSIR and QMDSCNN**: Authors of QMDSIR and QMDSCNN also tested the transferability of their datasets on DUC 2006 and 2007. The ROUGE-2 of their best performing model is **6.28** and **7.60** for DUC 2016 and DUC 2017 respectively. The ROUGE-2 for our LMGQS BART is **9.0** and **10.6**, significantly outperforming the model trained on QMDSIR and QMDSCNN. We will add the reference and empirical comparison to our revision.\n\nTo answer the \"Several unclear methods affecting readability and reproducibility\": \n\n**About the necessity to convert queries into a unified format** \n\nThe LMGQS dataset is designed to work with queries in the form of natural language questions. (see table 10 for query examples) However, different QFS datasets have different query formats. (table 3) For example, Debatepedia has the query format of a natural question, which is the same as LMGQS, while the majority of queries in DUC datasets are instructions such as “Discuss conditions on American Indian reservations or among Native American communities.” and “Include the benefits and drawbacks of the reservation system.” To use LMGQS in the zero-shot setting, it is necessary to convert these diverse query formats into natural language questions so that they can be processed by the LMGQS model. This process is called query unification.  \n\n**About the finetuning in query unification**\n\nWe propose to further utilize LMGQS for the query unification task by fine-tuning a BART model to generate queries with the document and summary as input. Basically, this is what Instruct-GPT was prompted to do in Section 2.1 of the paper. We input query of diverse format as pseudo \"summary\" and the generated query will be in a unified format. This approach allows for automatic conversion of diverse query formats into natural language questions that can be processed by the LMGQS model. The hyper-parameters for finetuning are the same as our main LMGQS BART model, which is described in section 3.1. We will make it clear in our revision. \n\n**About query template and examples**\n\nWe show them below for NEWTS and DUC datasets. And we will add a table showing this in our revision. \n- NEWTS \n  - Template: What are {topical words, phrases, or sentences}? \n  - Original query: winter temperatures, hot weather conditions, a cold morning \n  - Templated query: What are winter temperatures? What are hot weather conditions? What are a cold morning? \n- DUC:  \n  - Template: Replace the instructions such as 'describe', 'indicate', 'include', 'summarize' with 'What are the' \n  - Original query: Describe the activities of Morris Dees and the Southern Poverty Law Center. \n  - Templated query: What are the activities of Morris Dees and the Southern Poverty Law Center?", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* A new large scale dataset for QFS with more than 1.1M samples and a protocol for deriving it\n* Presenting the query unification task and providing a method for handling it", "reasons_to_reject": "1. It is desired to have more evidence or analysis supporting the training effectiveness property of the dataset or other key properties\nthat will explain the importance and possible use-cases of _LMGQS_ over other QFS datasets.\n\n2. Several unclear methods affecting readability and reproducibility:\n* \"To use LMGQS in the zero-shot setting, it is necessary to convert the queries of diverse formats into natural questions.\" (L350) please explain why.\n* \"Specifically, we finetune a BART model to generate queries with the document and summary as input.\" (L354) - how did you FT? what is the training set and any relevant hyper-parameters for reproducing the results.\n* \"we manually create a query template to transform the query into a natural language question.\" (L493) - what are the templates? what are the query template and several examples.", "questions_to_authors": "-", "paper_topic_and_main_contributions": "The paper presents a protocol to automatically derive a QFS dataset from a generic summarization dataset by utilizing generative models to generate questions for every summary sentence together with a new large QFS dataset (named _LMGQS_), by applying the protocol to 4 known summarization datasets. \nThe authors further fine-tune BART using _LMGQS_ and compare the effectiveness of the new dataset for training QFS models in the ZS and supervised settings.\nThe paper discusses the problem of unifying queries between different QFS datasets.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "* Figure 2 - hard to understand the exact ROUGE numbers, better have number labels on the bars or inline the results on the relevant section text."}, "extra_text": {"Missing_References": "The paper [Data Augmentation for Abstractive Query-Focused\nMulti-Document Summarization](https://ojs.aaai.org/index.php/AAAI/article/view/17611/17418) suggest data augmentation for MD-QFS and two automatically derived QFS datasets, _QMDSIR_ and _QMDSCNN_. It is desired to compare _LMGQS_ to them."}}, {"review_id": "RxBvuK8lKW", "reviewer": "Reviewer_Kn1x", "summary": "", "questions": "A. Could you elaborate on the rationale behind using the finetuned BART model (Gd,s_q) for query unification, as opposed to alternative methods? How did you determine that this approach would effectively convert queries of diverse formats into natural questions?\n\nB. Have you considered potential challenges or limitations in the unification process? Are there any cases where the generated queries might not align well with the intended natural question format?\n\nC. Given that InstructGPT generates queries based on human-readable instructions, have you investigated potential biases or controversial language that the model might produce? I suggest including a section on potential ethical concerns related to biased or inappropriate query generation.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer’s feedback! We will follow the review’s suggestions on the typo and presentation issues. \n\n**Question A** about the rationale behind using the finetuned BART model: We first addressed the query unification with human-designed templates (section 4.3). However, this method requires humans to read the raw queries and cannot be easily extended to unseen datasets. To automatically generate queries of a unified format, we utilize finetuned BART model (Gd,s_q) and input the raw query as \"pseudo\" summary. We manually checked the queries generated and most of them are of excellent quality.\n\n**Question B** about the potential challenges or limitations in the unification process: The potential challenge is that there is no way to directly assess the quality of the generated queries. We did observe a small number of generated queries that did not align well with the intended natural question format. However, most of them are of decent quality. And the end-to-end performance on the generated summaries could prove the effectiveness of our query uniform method, especially compared with the carefully handcrafted templated queries. (section 4.3)\n\n**Question C** about ethical concerns: We address the ethical concerns in the \"Ethics Meta Review\" of this page. And we will follow the reviewer's advice to include the discussion there in our revision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Innovative Dataset Construction. Paper introduces a creative approach to address the lack of large-scale query-focused summarization datasets. \n\n2. Effective Zero-shot and Supervised Performance. The proposed LMGQS BART model achieves sota zero-shot and supervised performance on multiple QFS benchmarks. This outcome underscores the potential of the methodology and dataset in enabling advancements in the development of QFS models.\n\n3. The paper extensively investigated the properties of the LMGQS dataset and conducted an evaluation of the proposed models against a range of baseline models. Human study also involved.\n\n4. Addressing Query Unification. The approach to query unification addresses a critical challenge in zero-shot QFS across diverse query formats. While there may be room for deeper analysis, the paper lays the foundation for a novel solution and demonstrates its superiority over handcrafted templates.", "reasons_to_reject": "1. The query unification approach for adapting diverse query formats to natural questions relies on finetuning a BART model. However, the paper does not thoroughly address the challenge of ensuring accurate and meaningful query conversions across different datasets. Ambiguities in query transformation may lead to misaligned summaries, impacting the overall quality of the model's outputs.\n2. There could be biases in the generated queries from InstructGPT.\n3. Insufficient discussion on the sensitivity of hyperparameters on final performance.\n", "questions_to_authors": "A. Could you elaborate on the rationale behind using the finetuned BART model (Gd,s_q) for query unification, as opposed to alternative methods? How did you determine that this approach would effectively convert queries of diverse formats into natural questions?\n\nB. Have you considered potential challenges or limitations in the unification process? Are there any cases where the generated queries might not align well with the intended natural question format?\n\nC. Given that InstructGPT generates queries based on human-readable instructions, have you investigated potential biases or controversial language that the model might produce? I suggest including a section on potential ethical concerns related to biased or inappropriate query generation.", "paper_topic_and_main_contributions": "This paper presents a novel approach to address the lack of large-scale datasets for query-focused summarization (QFS), a task that involves generating or extracting summaries from documents that directly answer or are relevant to specific queries. The paper proposes a solution to this challenge by leveraging a large-scale pretrained language model, InstructGPT, to generate hidden queries from existing generic summarization datasets. These hidden queries are then combined with document-summary pairs to create a new Query-focused Summarization Dataset. The main contributions of the paper are:\n\n1. Novel Dataset Construction Framework: by utilizing a language model as an annotator to generate hidden queries for existing generic summarization datasets, the authors create a new dataset without the need for labor-intensive human annotation of queries and summaries.\n2. LMGQS Dataset.\n3. Model Fine-Tuning and Performance: demonstrated the utility of LMGQS by fine-tuning a pretrained language model on it. The model takes the original document concatenated with the generated query as input and is trained to produce the original summary. The resulting model achieves state-of-the-art zero-shot and supervised performance on multiple existing QFS benchmarks.\n\nIn summary, this paper addresses the challenge of creating large-scale datasets for query-focused summarization by leveraging a language model to generate hidden queries. ", "excitement": 4, "reproducibility": 4, "ethical_concerns": "Yes", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "section 3.4 last paragraph: \"quires\" -> \"queries\""}, "extra_text": {"Justification_for_Ethical_Concerns": "Given that InstructGPT is a pre-trained model that generates queries based on human-readable instructions, it's unclear whether/how the authors investigated potential biases or controversial language that the model might produce. There could be potential ethical concerns related to biased or inappropriate query generation."}}, {"review_id": "eoytylE6qG", "reviewer": "Reviewer_udAj", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer’s feedback! Regarding the quality of the generated queries in LMGQS, we have verified their quality through zero-shot and supervised performance on several query-focused summarization datasets. The state-of-the-art performance of LMGQS confirms the high quality of the queries generated. This is evidenced by the fact that the BART model, fine-tuned on LMGQS, can accurately capture the correspondence between the input query and the relevant summary. Furthermore, the performance gain of LMGQS BART over other strong baselines further confirms its superior quality.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Interesting way to augment data\n", "reasons_to_reject": "The key novelty seems to be the use of LLMs to augment the queries. However, the quality of the queries thus generated is not validated much - the qualitative evaluation seems to look at the ability of the trained models to match the reference in datasets. I would have loved to see more detailed analysis here. ", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper proposes a novel approach to curate a large dataset for query-focused summarization, Using Large Language Models, queries are constructed and augments to existing summarization datasets - which is then used to train a large model for a generalized QFS task; improvements shown across various QFS datasets. ", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Considering ethics issues pointed by  Ethics Chairs, I would suggest rejection of this paper.|Ethical concerns seemed to have been resolved by authors and ethics committee (only acknowledgement was needed). Reviewers seem to agree that this work's proposed methods are novel and have merits. Some concerns were still flagged but overall they were positive.
Decision: Accept-Findings