Reinforcement Replaces Supervision: Query focused Summarization using Deep Reinforcement Learning
Authors: Swaroop Nath
Pushpak Bhattacharyya
Harshad Khadilkar
Venue: EMNLP 2023 Main (2023)
Keywords: reinforcement learning for long text
query focused summarization
passage embedding
long form question answering
TL;DR: New RL algorithm for long text generation in Query-focused Summarization. Proved to be effective on benchmark datasets and new gold standard data. Also introduced a novel Passage Embedding model and released pre-training dataset.

Abstract: Query-focused Summarization (QfS) deals with systems that generate summaries from document(s) based on a query. Motivated by the insight that Reinforcement Learning (RL) provides a generalization to Supervised Learning (SL) for Natural Language Generation, and thereby performs better (empirically) than SL, we use an RL-based approach for this task of QfS. Additionally, we also resolve the conflict of employing RL in Transformers with Teacher Forcing. We develop multiple Policy Gradient networks, trained on various reward signals: ROUGE, BLEU, and Semantic Similarity, which lead to a $\mathit{10}$-point improvement over the State-of-the-Art approach on the ROUGE-L metric for a benchmark dataset (ELI5). We also show performance of our approach in zero-shot setting for another benchmark dataset (DebatePedia) -- our approach leads to results comparable to baselines, which were specifically trained on DebatePedia. To aid the RL training, we propose a better semantic similarity reward, enabled by a novel Passage Embedding scheme developed using Cluster Hypothesis. Lastly, we contribute a gold-standard test dataset to further research in QfS and Long-form Question Answering (LfQA).

Reinforcement Replaces Supervision: Query focused Summarization using
Deep Reinforcement Learning
Swaroop Nath Pushpak Bhattacharyya* Harshad Khadilkar
* Computer Science and Engineering, IIT Bombay, India TCC Rescarch, India
(swaroopnath, pb]@cse. ii ac. harshad, mhadilkar@tcs.oo com
Abstract
Query: What is String Pool in Java?
Query- focrusfo Summarization (QfS) deals with systems that generate summaries from document(s) based on query. Motivated by the insight that Reinforcement Learning (RL) provides a generalization to Supervised Learn- ing (SL) for Natural Language Generation, and thereby performs better (empirically) than SI,, we use an RL-based approach for this task of QrS. Additionally, we also resolve the conflict of employing RL in Transformers with Teacher Forcing. We develop multiple Policy Gradi- ent networks, trained various reward sig- nals: ROUGE, BLEU, and Semantic Similar- ity, which lead point improvement over the State-of-the-Ar approach on the ROUGE-L, metric for a benchmark dataset (ELI5). We also show performance of our approach in zero-shot setting for another benchmark dataset (Debate- Pedia) - our approach leads to results com- parable baselines, which were specifically trained on DebatePedia. To aid the RL training, we propose a better semantic similarity reward, enabled by novel Passage Embedding scheme developed using Cluster Hypothesis. Lastly, we contribute gold-standare test dataser to fur- ther research in QfS and Long-form Question Answering (LfQA).
Document String pool is nothing but stor- age area in Java heap where string literals stores. It is also known as String Intern Pool or String Constant Pool. It is just like object al- location. By default,
empty and privately maintained by the Juva ...
Summary: String Pool, used reduce the memory footprint, is specific area in the memory allocated to the process used to store String literal declared within Java program.
Table 1: Example for Query-focused Summarization for well-formed question. The document is shortened (marked by ...) for space.
Problem Statement: We develop a system which abstractively generates summaries (Nallapati et al., 2016; Rush et al., 2015; Xu and Lapata, 2021, 2022) from single document given the query. Specifically, input: query and document, and out- put: summary. We design a novel Reinforcement Learning algorithm for training and propose novel passage embedding- based reward function.
Motivation: QfS lets the user drive the summariza- tion. This improves the user experience by letting the user extract the needed information quickly. Additionally, present-day QA systems (Calijorne Soares and Parreiras, 2020) produce only short an- swers to factual questions. What they lack is the ability to consume a large piece of information and present coherent, compressed summary based on the needs of the user. Our work aims further research in LfQA by successfully building QîS system for questions.
1 Introduction
Query-focused Summarization (QIS) (Tombros and Sanderson, 1998; Dang, 2005; Nema et al., 2017) advances text summarization by letting the user provide a query, pertaining to which the sum- mary must be generated from the document(s). Specifically, we target QfS for questions on a single document (see Table for example). Our work is related to Long-form Question Answering (LfQA) (Fan al., 2019; Krishna et al., 2021; Su et al., 2022). However, it differs from LfQA research significantly in that LíQA research has to focus on the retrieval of relevant passage(s) also. Whereas our system already assumes the presence of doc- ument, which has to be summarized.
We utilize Reinforcement Learning (RL) for training the model, as RL provides generalized loss to train our model. While Cross-Entropy loss
Source: javatpoint www.:/iwn...oottinnnnneeeeeeeeee -ool-in-jaaa

---

utillizes loken-by-token match, RL helps us general- ize this further phrase/sentenccesemaniii match. Such generalization gives the model more free- dom on what to generate at each time step, as long as the final generation suffices. Inspired by this, we employ RL for QIS. We detail on how RL is generalization to Supervised Learning (SL) using Cross-Entropy loss in Section 3.1.
2 Related Works
Supervised Learning ap- proaches to abstractive Q have been primarily limited by the availability of large-scale datascts. Dang (2005) present the first QPS dataset with open-ended questions. However, the size of the dataset is insufficient to train neural models. Xu and Lapata (2021, 2022); Laskar et al. (2020) tackle the scareity of data through innovative approaches. Xu and Lapata (2021) generate proxy queries from generic summarization datasets by using a masked representation to train model for QfS. Xu and Lapata (2022) model the document as an indicator of all the possible queries on it, and develop a QÍS model by modeling latent queries through variational inference techniques. Laskar et al. (2020) take the approach of Transfer Learning: they fine-tune an abstractive summarizer, trained on XSum (Narayan et al., 2018), on a decent-sized (12695 instances) dataset: DebatePe- dia (Nema et al., 2017). Inspite the presence of DebatePedia, researchers have mainly focused on leveraging generic summarization datasets due to the size limitation. We circumvent this limitation by utilizing ELI5 (Fan et al., 2019). Specificially, utilize the version by Fan et al. (2019), not the KILT version (Petroni al., 2021), as the latter does not provide gold documents to generate the summary from. Although, it is dataset for LIQA, the format of the dataset makes suitable for our use case: QSS for question on a single document. EL15 is an automatically curated dataset with few shortcomings, which make it unsuitable for testing models (Krishna al., 2021). Motivated by these shortcomings, we propose a novel, human- curated test set for QíS on well-formed questions consisting of high-quality 250 instances.
A Abstractive QIS:
However, employing RL for text generation ing Transformers is non-trivial. RL helps train agents that perform a sequence of actions, each de- ciding the next state and the available set of actions. In contemporary text generation through Trans- formers, Teacher Forcing" (Williams and Zipser, 1989) is used, where the generation (action) at time- step 1 has influence the generation at time-step t. Effective utilization of RL for text gen- eration needs model this influence, thus necessi- tating the omission of Teacher Foreing. However, this increases the training time and memory foot- print significantly. We propose a way to employ RL for text generation without Teacher Forcing, based on Scheduled Sampling (Bengio et al., 2015), Sec- tion 3.1. To the best of our knowledge, we are the first work to resolve this conffict of employing RL in Transformers with Teacher Forcing. Our contributions are:
1. An RL algorithm that resolves the conflict of RL-based training in Transformers with Teacher Forcing. We observe significant im- provement over the State-of-the-Are SL mod- cls: 5% improvement in automatic evalua- tions (Table 6), 19% improvement in Correct- ness (human evaluation; Table 8).
2. A human-curated test set, 250 instances, de- void of Topic Centralization (one document can cater to multiple queries, Baumel et al. (2016)), for analysis of QIS models.
RL for Summarization/Q(S. The usage of RL in QIS has been limited to extractive summariza- tion only (Mollá and Jones, 2019; Mollá al., 2020; Chali and Mahmud, 2021; Shapira et al.. 2022). Mollá and Jones (2019); Mollá et al. (2020); Shapira et al. (2022) use RL to train sentence selec- tor models, which select sentences to be incorpo- rated into the summary. Chali and Mahmud (2021) present a hybrid summarization, where extrac- tive module selects text from the document, which is then used by the abstractive module gener- ate an abstractive summary. However, they use RL for the extractive module only. To the best of our knowledge, we are the first to utilize RL for
3. A passage embedding based novel reward mechanism (2.21 ROUGE-L improvement over ROUGE reward; Table 6) score gen- erated summaries, trained using the long- standing, time-honored Cluster Hypothesis (Jardine and van Rijsbergen, 1971).
4. A new dataset, 8 million instances, to train a passage embedder model, seraped from reddit.
Teacher Forcing refers to the way of training Autoregres- sive models where generation of yt (current token) depends on yI, y2, -.., 1 (true previous tokens), and not on ÿI, ÿ2, it-1 (generated previous tokens).

---

abstractive QfS.
3 Modeling
In this section we provide the formulation for Q and Passage Embedding. Section 3.1 discusses the application of Policy Gradient to QTS. Section 3.2 discusses our approach to obtain passage embed- dings. We present architectures in Appendix G.
We find abundance of precedence of RL for abstractive summarization. Paulus al. (2017) is the first work in employing RL to train LSTM models, without Teacher Forcing, for abstractive summarization. Pasunuru and Bansal (2018); Li et al. (2019) provide follow-up works with better reward functions. Both works train LSTM models without Teacher Forcing. Laban et al. (2020) were the first to employ RL to Transformer architec- ture for abstractive summarization. They fine-tune the GPT2-small (Radford et al., 2019) model with- out Teacher Forcing. However, the omission of Teacher Forcing led to a training time of 10 days for a generation length of just 10 tokens. This highlights the severity of the problem which arises by omitting Teacher Forcing and using sequential generation during training, and motivates the need for our approach effectively incorporate RL.
3.1 Policy Gradient for QfS
We model the task using Reinforcement Learning (RL) framework. Reward Hypothesis (Sutton and Barto, 2018: Silver al., 2021) states that all goals can be formulated the maximization of reward signal. Following this, our goal is to represent QíS using a reward adequately. Equations and specify the losses from Maximum Likelihood Estimation (MLE; used by Supervised Learning through cross-entropy loss) and Policy Gradient training. Comparing the two equations, it is vis- ible that MLE attempts to reward token-by- token match We take the RL path generalize this rewarding mechanism using more sophisti- cated lexical and/or semantic rewards (Table 2). Terminology: [xx] evaluates to 1 if x is true clse 0, v denotes the token generated at time-step k. is the trajectory obtained from the sequence of token generation, and d denote the query and document representation respectively, and n is the length of the generation.
Passage Embeedding/Simiilrity:: A straight- forward way to obtain paragraph/ paassgge embeddings by application of some composition- ality to the embeddings of the constituent words (Kenter et al., 2016; Hill et al., 2016; Sinoara et al., 2019; lyyer et al., 2015). Yang et al. (2020); Jiang et al. (2019) attempt to generate passage embedding by training dual encoder networks to match similar documents. They rely citation and recommendation networks to derive whether or not two passages are similar. Obtaining passage similarity/cmbeeldnng is a common task in Dense Passage Retrieval (DPR) (Karpukhin et al., 2020). Ren et al. (2021) utilize Cluster Hypothesis (Jardine and van Rijsbergen, 1971) to enhance passage retrieval. However, they do not train any passage embedding network separately. Ginzburg al. (2021) utilize pairwise sentence similarity to obtain similarity between two paragraphs. They use a noisy training dataset where paragraphs from the same document are considered similar and those from different documents are considered dissimilar. We are primarily motivated by the work of Vikraman et al. (2021). They use Cluster Hypothesis group similar passages together in the DPR framework, for downstream tasks. Our work differs from Vikraman et al. (2021) in the following aspects: we explicitly train our model to output passage embeddings and we train on a much larger dataset.
Σ [ve = =v log P(vi Puvilvi, vii
LMCE =
,. d.) (1)
n
Σ { (R(T) - b) log P(y
Lpg =
vi- 9.
(2)
We formulate QSS in the RL framework (<S, A. R, P> tuple) as follows:
The agent (encoder-deconl) network) exists a state (S), defined by the thus-far gener- ated summary, the input query and the input document.
2. It takes action (A) by sampling token (v)) from the probability over the vocabulary for the next time-step. The generation of probabil- ity distribution constitutes the state transition dynamics (P).

---

3. The end of the episode (sequence of genera- tion) is reached either through the generation of end-of-sequence token or by reaching the max episodic length.
training, using the cosine similarity between the generated passage vectors. We find that similarity obtained from this embedding generation scheme leads to better results in human and automatic eval- uation (Tables 6 and 8).
4. At the end of the episode the agent achieves reward (R), which then used by the Pol- icy Gradient algorithm (Equation 2) to update the policy for the agent. We use the greedily decoded sequence to compute the baseline re- ward (b in Equation 2). following Paulus et al. (2017).
We use the Cluster Hypothesis for training our passage embedding model, using dual encoder architecture (Appendix G). Our training scheme considers special token (for example, [CIS] in BERT) to represent the passage. We calculate the similarity, J. between passages, using the dot prod- uct of the embeddings (Ep and Eq) produced by the dual encoder architecture, Equation 4. This causes similar Ep and Eq to lie closer in the embedding space. While training, we attempt to reduce the cross-entropy loss, Equation 5, between ý and y (the true labels; 1 for similar passages, else 0).
However, it is intractable to run this vanilla RL framework for long episodes (~ 150 tokens, Ap- pendix G. Table 10) when the policy has a huge number of parameters. It leads to large time for forward and backward pass through the network, in addition the large memory requirements. Re- fer to Appendix E for a space and time complex- ity analysis. We simulate the sequential action sampling through scheduled sampling for Trans- formers (Mihaylova and Martins, 2019). Following the strategy, we conduct two-pass through the decoder: the second pass uses the sampled embed- dings, obtained using Gumbel reparametrization trick (Goyal et al., 2017). This enables gradient backpropagation for the first pass through the de- coder too. We find that using an implementation where gradients backpropagate through both passes leads to better results in automatic evaluation (Ta- ble 6). With this formulation, we train the policy network using mixed objective loss (Equaaiinn 3, 17 0.1), following Paulus et al. (2017).
" I Ep
(4)
Cpg -ee--ylogg (1 y) log
(5)
We use the trained passage embedding model to obtain embeddings for the generated summaries and the ground truth summaries training the Q model. We use cosine similarity as a reward signal the RL framework.
batch while
Datasets
We use three datasets in this work: (a) EL15 (Fan et al., 2019), (b) Reliable QFS Tester, RQFT (our contribution), and (c) Reddit Passage Embedding DataseT, RPEDT (our contribution). In the fol- lowing discussion, we provide brief overview of the EL15 dataset, and move on to discuss RQFT in Section 4. 1, and RPEDT in Section 4.2.
LTOTAC = m/C.CE + (1 Crg
3.2 Passage Embedding and Semantic Similarity
Fan et al. (2019) proposed ELI5 for Long- form Question Answering (LfQA). It consists of a triplets in the format: <query, document, answer>, where the query is well-formed, open-ended ques- tion, and the answer is not span from the docu- ment. Given the nature of the query and the an- swer, it fits perfectly in our problem statement. The dataset contains 234420, 9930 and 24820 samples for training, validation and testing respectively.
Contemporary works rely on composing sen- tence/word embeddings to obtain passage embed- dings. This forms an approximate representation of the generated and the ground truth summary. Thus, relying on these representations leads to an approximate notion of similarity. We amend that by creating passage embedder using the Cluster Hypothesis (Jardine and van Rijshergen, 1971), which states: Passages that are clustered together answer similar information needs. According to the hypothesis, summaries of the same query on the same document should be clustered together and have a high similarity score. We use this insight to reward the generated summaries while
4.1 Reliable QFS Tester (RQFT) Dataset
We note two shortcomings of the ELI5 dataset: Overlap of instances between training and val- idation set, reported by Krishna et al. (2021).
2. <query, document, answer> triplets in the ELI5 test set, where the document is irrelevant

---

Reward
Description
Formula
ROUGE-L
Recall oriented reward to improve coverage
PROUGEL(GT GN)
BLEU
Precision oriented reward to generate con- cise summaries
mean (BLEU, (GT,GN)
SimCSE (Gao et al., 2021)
Semantic match obtained by averaging
cos(mean,, mean, (ET)
tence embeddings
SBERT (Reimers and Gurevych, 2019) SFPEG
Semantic match obtained by averaging sen-
cos(meanm (Es)) mean, (ET))
tence embeddings
Semantic match obtained using Passage cos(ECT, EGN) Embedding
Table 2: Various rewards used train the RL models. We use both lexical and semantic rewards promote lexical and semantic similarity in generation. SFPEG: Similarity From Passage EmbeddinG, ROUGE, and BLEU denote the standard ROUGE-L and BLEU functions; mean denotes average; cos denotes cosine similarity; $ denotes a sentence from Ground Truth (GT) summary; denotes a sentence from Generated (GN) summary.
to the query. We observed this shortcoming while analyzing the trained models on random samples from the test set.
Characteristic
Value
Size of dataset
250
Query
15.52
Motivated by these, we curate a dataset with manual efforts, without any automation scripts, to ensure quality. We curate the dataset from two sources- (i) Wikipedia and (ii) high school text- books. Both excellent sources for variety of concepts. We explain how the dataset curated in Appendix K.
Avg. of words
Document
930.57
Summary
115.72
Avg. # of queries per document
1.41
Table 3: Statistics of RQFT dataset, we use NLTK to tokenize the strings.
Table 3 presents statistics on the dataset. We can see that each document corresponds to more than 1 query average. This was an intentional de- cision made while curating the dataset to tackle topic centralization (Baumel al., 2016). We include an example of the dataset in the Appendix (Table 17). We establish the quality of the dataset in Appendix
contain data in the format of posts (queries) and comments (answers to the posts, often multiple in
number). We scrape this data, forming repository of queries and multiple answers each query. We gather data from 39 subreddits, Appendix F de- scribes the data gathering scheme and lists the sub- reddits. While training, we transform the dataset into the following format- <p, q>, where p and are passages that may or may not answer the same query. We perform in-batch negative random sampling generate q (one q per p) that does not answer the same query as p. Table highlights the statisties; the last row denotes the total number of <p, q> samples generated during training. Our quality check experiment, Appendix M, validates the quality of RPEDT.
While the size of RQFT is small, such a size for evaluation is not unprecedented for summarization (Angelidis and Lapata, 2018; Chu and Liu, 2019; Bražinskas al., 2020; Amplayo al., 2021; An- gelidis et al., 2021). Inspite of the small size, our dataset acts as a high quality benchmark for QíS, covering domains (listed in Appendix K), de- void of topic centralization.
4.2 Reddit Passage Embedding DataseT (RPEDT)
Experiments and Results
Cluster Hypothesis states: Passages that clus- tered together answer similar information needs.
In this section provide details experi- ments. Section 5.1 presents the results obtained on all the experiments. Finally, Section 5.2 presents the results obtained from human evaluation on RQFT. Table 5 lists the trained models. BART
We gather RPEDT facilitate the training of passage embedding model using Cluster Hypothe- sis. Reddit forums (popularly known as subreddits)

---

Value
tokens 64, maximum tokens 256). We find that using these generation conditions work best (based on automatic evaluation). We present the results generated for all the experiments in Table 6. We also present the average length of generations in Table 7. Using these generation parameters, we see that the BART SL model obtains better ROUGE- L (1.14 points) than the one presented by Lewis et al. (2020). We attribute this gain to the usage of Scheduled Sampling, leading more robustness in generation than a model trained using Teacher Forcing.
Characteristic
Train
Test
# of questions
154, 722
19, 647
Avg. # of A per Q
3.04
2.26
Avg. # of W in Q
15.58
17.80
Avg. # of W in A Training samples
145.36
170.88
15,462, 462, 880
223, 046
Table 4: Statistics of RPEDT, we use NLTK to tokenize into words. Q: Question, A: Answer, W: Words.
SL represents our implementation of the model trained by Lewis et al. (2020) for ELI5.
We can see in Table that the models trained using Reinforcement Learning (RL) perform sig- nificantly better than BART SL (the baseline), with 10.62 improvement on ROUGE-L for ELI5 test set. We also see that the models obtain signifi- cantly better results, than the baseline, on our test dataset (ExPT-II), achieving as high as 14.89 im- provement on ROUGE-L. The RL-based models achieve better scores on our dataset as compared to ELJ5 test set, however, we fail to see such sig- nificant gains for BART SL. We present possible reason for this in Section 6.3. Also, we see that the scores for EXPT-II and EXPT-III are closer for BART SL, in comparison the RL-based models. This indicates two things-
Model ID
Reward
BART SL
BART R
ROUGE-L
BART R-SEM
ROUGE-L + SimCSE
BART R-B
ROUGE-L + BLEU
BART R-SBERT
ROUGE-L SBERT
BART R-SEPEG
ROUGE-L + SEPEG
BART SEPEG
SEPEG
Table 5: Index of all the models trained in our work. All models except BART SL have been trained using Reinforcement Learning: the rewards are listed the adjacent column. BART SL has been trained using Cross Entropy loss (Supervised Learning).
1. BART-SL model generates relevant (to the query) content (irrespective of the document- random or true). But, the RL models actually utilize the provided document. This shows that the RL models truly learn the task of QfS, justifying the significant boost (~ 10 points).
5.1 Automatic Evaluation Results
We perform three experiments- (i) EXPT-I: QfS on ELI5 test dataset, (ii) EXPT-II: Q&S on RQFT, and (it) EXPT-III: QíS on RQFT with random docu- ments. Additionally, we also report automatic eval- uation figures on DedatePedia (Nema et al., 2017). As the training and primary analysis involves only ELI5, move the DebatePedia results to the Ap- pendix (Appendix D). In EXPI-III, we replace the true document with another random document, The motivation behind this experiment was to rig- orously test whether the trained models summarize the document based on the query, or generate to- kens by ignoring the document altogether. We use the Fan et al. (2019) version of ELI5 dataset, not the KILT version. Hence present competing re- sults from Fan et al. (2019) and Lewis et al. (2020) only, and omit results on KILT version, such as those from Su et al. (2022).
2. While Krishna et al. (2021) point out several shortcomings of the ELI5 dataset, it can be seen that, using the dataset, RI. models can learn to summarize according query given the true document.
5.2 Human Evaluation Results
We present human evaluations on Fluency and Cor- rectness our models. We use two annotators for human evaluation. We ask them to rate the Flu- ency on Likert Scale ("Very Poor", 0 to "Very Good", 4). For Correctness, we follow YES (1) or NO (0) marking scheme. We randomly sample 50 instances from the RQFT dataset, and present the generated summaries, and the human written summaries the annotators. The summaries are anonymized, such that the evaluators have no un- derstanding of the source: human written or auto-
We test the performance of the trained models using Beam Search (beam size = 15, minimum

---

ELI5 dataset
Our dataset (EXPT-II)
Our dataset (EXPT-III)
Model
R-1
R-2
R-L
R-1
R-2
R-L
R-1
R-2
R-L
Fan et al. (2019)
28.9
5,4
23.10
28.82
6.97
25.41
23.13
4.39
20.91
Lewis et al. (2020)
30.6
6.2
24.3
BART SL (b)
29.68
5.89
25.44
29.67
7.88
26.40
23.32
4.51
20.96
BARTH R
38.93
8.05
34.54
43.08
15.30
39.08
24.38
4.40
22.17
BART R-SEM
38.02
6.56
33.13
44.12
15.15
40.39
25.39
4.51
23.06
BART R-B
39.52
8.25
34.92
13.46
15.67
39.29
26.01
1.88
23.45
BART R-SBERT
36.9
6.36
32.8
42.93
15.10
39.56
25.41
4.47
23.19
BART R-SEPEG
39.40
6.92
34.10
45.52
16.83
41.29
25.89
4.99
23.50
BART SHPEG
34.76
5.96
29.66
44.30
15.79
40.39
25.82
4.63
23.61
Table 6: Quantitative comparison of our models. For the purposes of comparison, we also provide the results obtained by Fan (2019) and Lewis et al. (2020). We use ROUGE metrics, ROUGE-1 (R-1), ROUGE-2 (R-2), and ROUGE-L (R-L) to compare the models on both the ELI5 test dataset (EXPT-I) and RQFT (EXPT-II: with true document and EXPT-III: with random document). We use BART SL as the baseline (denoted by b) to judge the efficacy of training models under the Reinforcement Learning framework. BART SL is our implementation of Lewis et al. (2020), hence row 3 represents the results for Lewis et al. (2020) for EXPT-I1 and EXPT-III.
Model
EXPT-I
EXPT-II
R-SEPEG model is significantly better than the BART SL model. We find that BART R-SFPEG is significantly more correct than BART MLE, with a 10% significance level (p colue = 0.077). In terms of Fluency, BART R-SIPEG is significantly more fluent than BART SL model with 1% sig- inificance level (p value 0.00012).
BART SL
77.32
95.78
BART R
234.29
234.02
BART R-SEM
239.98
233.84
BART R-B
241.27
235.89
BART R-SBERT
237.09
235.19
BART R-SFPEG
239.23
232.73
BART SHPEG
239.19
234.93
Model
Fluency
Correctness
Table 7: Length of generations (# of words) from the models. We use NLTK to tokenize the generations. We provide an analysis on how RL models utilize more tokens for better summarize in Section 6.1.
BART SL
3.08 3.12
0.26 / 0.28
BART R
3.34/ 3.30
0.38 0.32
BART R-SEM
0.24 / 0.18
BART R-B
3.24/3.22
0.28 / 0.26
BART R-SEM
/.227
0.22 / 0.26
matically generated or even the generator model. Table 8 presents the results generated from human evaluation. We can see similar trend as in Ta- ble 6: BART R-SIPEG model obtains the highest Fluency and Correctness.
BART R-SFPEG
3.46 3.48
0.48 / 0.44
BART SFPEG
3.06
0.28 / 0.26
Human
4.99
1.0 1.0
Table 8: Results from human evaluations of the machine generated text. The scores reported in A format; where A is the average score computed from ratings by Annotator and B the average score computed from ratings by Annotator 2.
We also compute the IAA scores for Fluency and Correctness for the evaluations of the models. For Fluency, we map the scores to ratings: Pos- itive (3 and 1), Neutral (2) and Negative (0 and 1), and compute the fraction of times the anno- tators agree on the ratings. We observe that the annotators agree almost always, with minimum agreement of 0.88 (for BART SL). For Correctness, we use Cohen Kappa score to report the agreement, We observe moderate agreement (McHugh, 2012). with a minimum score of 0.61 (for BART R-SEM). Through hypothesis testing, we validate that BART
Analysis
Our initial analyses the EL15 test set revealed recurring problem: our models (all of the 7 models) would copy the query into the generated summary. On further probing, we discovered that this cor- related highly with the absence of content the

---

document relevant to the given query. Frequent occurrences motivated to create a manually cu- rated test set for reliable analyses. In this section we present a detailed analysis of our models. In Section 6.1 we present a comparative analysis of all the models, in Section 6.2 we analyse the mod- els when multiple queries are posed on the same document and in Section 6.3 we analyse our mod- els when the document is replaced with random document. Our analysis reveals that the RL models actually learn QiS as compared the BART-SL model, which has the tendency to ignore the pro- vided document. This justifies the significant boost (~> 10 points) in performance.
manages to generate different summaries for the two queries, they are either hallucinated or mostly unrelated to the query. Whereas, the RL-based
model manages to understand the necessities of the queries, and generates summaries for them.
6.3 Analysis for Random Document
Section 5.1 motivates the need for this experiment. Quantitative results show that BART SL does not have significant drop in performance the Reinforcement Learning (RL) based models when we use random document. While analyzing the generations, we observe that the BART SL model ignores the document. Although the generated text stays relevant to the query, it cannot be stated as query focused summary of the document, as the content is absent in the document. This leads us to believe that the BART SL does not truly learn to solve QfS, which also explains the relatively insignificant change in scores when the quality of the dataset is improved (ELI5 test set vs RQFT, Table 6).
6.1 Comparative Analysis
All the Reinforcement Learning (RL) based models generate self-contained,, easy to follow summaries. On the other hand, the BART SL model generates very erisp summaries, too frugal with the amount of words (Table 7). The RL-based models present a lot of relevant material in the generated sum- mary, but the BART SL model jumps straight to the point. While the latter can be usually desired, the tendency to be too frugal with words can leave the user wanting for more. We include an exam- ple in Appendix A to highlight the differences. We believe that this difference is result of using ROUGE as reward signal. ROUGE leads the model to choose more of the relevant content while training to increase the reward. However, we also observe a downside to this: as ROUGE rewards same word stems, we see multiple tokens with the same stem, discussed in Appendix A. Although the BART-SFPEG model does not use ROUGE as reward, we see that it also has verbose genera- tions. This is also understandahle: choosing more relevant content increases semantic match too. We also note that BART SL model hallucinates (Ji et al., 2022) significantly more than all the RL- based models, inspite of the crispness, which con- tributes to lower Correctness score (Table 8). We highlight this phenomenon in Appendix A.
We include an example of generation from ran- dom document for BART SL and an RL-based model in the Appendix C. We see that the RL- based model generates content from the document, based on whatever understands from the query, which strongly confirms our belief that the RL- based models learn much better solution to QfS.
Conclusion and Future Work
In this work we present Reinforcement Learning (RL) based models for the task of QES. We observe that these models perform significantly better than a model, with the same architecture, trained using Supervised Learning (SL). We also observe that the RL-based models generate summaries with much lesser hallucination than the Supervised Learning model. Additionally, also resolve the conflict of employing RL in Transformers with Teacher Fore- ing, by utilizing Scheduled Sampling. We present a novel reward derived using the Cluster Hypothesis. Through the work, we also contribute two datasets for the community: RPEDT, to train passage em- bedding models, and RQFT, a gold standard test dataset for analyses of QIS models.
6.2 Multiple Queries Same Document
We also test abilities of the models when they are probed with different queries on the same docu- ment. We observe that the Reinforcement Learning (RL) based models perform better in this arena too. Appendix includes an example highlighting the difference between the BART SL and an RL-based model. We see that, although the BART SL model
The takeaway from our work is that RL is better framework tackle QTS. We observe that RL helps the model learn QfS much better than SL, even with straightforward lexical rewards, such as ROUGE. Additionally, we also conclude, from results, that

---

Cluster Hypothesis leads much better semantic fcedback than competing passage embedders. Training RL models for long horizon (higher generation length) poses challenges in terms of ex- ploration of action space and temporal credit/blame assignment (Jiang and Agarwal, 2018). In our fu- ture work, we would focus on tackling this chal- lenge. Our motivation is that it would reduce the convergence time, and provide even better results.
References
Reinald Kim Amplayo, Stefanos Angelidis, and Mirella Lapata. 2021. Aspect-controllable opinion summa- rization. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. pages 6578 6593,, Online and Punta Cana, Domini- can Republic, Association for Computational Lin- guistics.
Stefanos Angelidis, Reinald Kim Amplayo, Yoshihiko Suhara, Xiaolan Wang, and Mirella Lapata. 2021. Extractive opinion summarization in quantized trans- former spaces. Transactions the Association for Computational Linguistics, 9:277-293.
8 Ethics Statement
We present two new datasets in our work: RPEDT and RQFT. While curating RQFT, we attempt to refrain from including any sensitive topic into our dataset, based on the knowledge of the curators and the quality check annotators. However, we cannot guarantee such claims for RPEDT. We understand that the passage embedding models trained on the dataset can learn biases and stereotypes present in RPEDT. Hence, we urge the community to use our passage embedding models and RPEDT with caution.
Stefanos Angelidis and Mirella Lapata. 2018. Sum- marizing opinions: Aspect extraction meets senti- ment prediction and they are both weakly supervised. In Proceedings of the 2018 Conference Empiri- cal Methods in Natural Language Processing, pages 3675-3688 Brussels, Belgium. Association for Com- putational Linguistics.
Tal Baumel, Raphael Cohen, and Michael Elhadad. 2016. Topic concentration in query focused summa- rization datasets. In AAAI Conference on Artificial Intelligence.
We utilize a total of 4 annotators to conduct qual- ity check experiments and human evaluation. 2 annotators are post-gramt post-graduates and the other 2 are under-graduates in the Computer Science depart- ment of an organization (medium of communica- tion: English). The annotators have been paid suffi- ciently, according to the geographical location, on an agreed upon rate.
Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. 2015. Scheduled sampling for sequence prediction with recurrent neural networks. In Pro- ceedings of the 28th International Conference on Neural Information Processing Systems Volume 1, NIPS' 15, page 1171-1179, Cambridge, MA, USA. MIT Press.
Arthur Bražinskas, Mirella Lapata, and Ivan Titov. 2020. Unsupervised opinion summarization as copycat- review generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Lin- guistics, pages 5151-5169, Online. Association for Computational Linguistics.
9 Limitations
The most striking limitation of our work is the necessity of huge computation power, which has restricted us from an extensive experimentation on hyperparameter search. In light of such limitations, we report results from a single run. And we also acknowledge the presence of hyperparameter con- figurations that can lead to better performance of the models. However, we note that such limi- tation does not mirror during inference, and our models can be chcaply (relatively) deployed for public use.
Marco Antonio Calijorne Soares and Fernando Silva Parreiras. 2020. A literature review on question an- swering techniques, paradigms and systems. Journal of King Saud University- Computer and Information Sciences, 62656663.
Yllias Chali and Asif Mahmud. 2021. Query-based sum- marization using reinforeement learning and trans- former model. In 2021 IEEE 33rd International Con- ference on Tools with Artificial Intelligence (ICTAT), pages 129-136.
Another limitation of our work is that the pas- sage embedding models have been trained on rela- tively limited data gathered from only source (Reddit). We acknowledge the fact that training on varied sources can lead to better representation of passages, however, keeping time and resource constraints in mind, we train our models on data gathered from only one source.
Eric Chu and Peter J. Liu. 2019. Meansum: A neural model for unsupervised multi-document abstractive summarization. JCML.
Hoa Trang Dang. 2005. Overview of due 2005. In Proceedings of the Document Understanding Conf. Wksp. 2005 (DUC 2005) the Human Language Technology /Cnf. on Empirical Methods Nut- ural Language Processing (HLT/EMMLP..

---

Jacob Devlin, Ming-Wor Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Tech- nologies, Volume 1 (Log and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Jyun- Yu Jiang, Mingyang Zhang, Cheng Li. Mike Ben- dersky, Nadav Golbandi, and Mare Najork. 2019. Semantic text matching for long-form documents. In Proceedings of the 20/9 World Wide Web Conference, pages 795-806.
Nan Jiang and Alckh Agarwal. 2018. Open problem: The dependence of sample complexity lower bounds on planning horizon. Annual Conference Compu- tational Learning Theory.
Angela Fan, Yacine Jernite, Ethan Perez, David Grang- ier, Jason Weston, and Michael Auli. 2019. Eli5: Long form question answering. In Proceedings of ACI. 2019.
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open- domain question answering. In Proceedings of the 2020 Conference on Empirical Methods Natural Language Processing (EMNLP), pages 9769-6781 Online. Association for Computational Linguistics.
Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021. SimCSE: Simple contrastive learning of sentence embeddings. In Empirical Methods in Natural Lan- guage Processing (EMNLP).
Tom Kenter, Alexey Borisov, and Maarten de Rijke. 2016. Siamese CBOW: Optimizing word embed- dings for sentence representations. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 941-951, Berlin, Germany. Association for Computational Linguistics.
Dvir Ginzburg, Itzik Malkiel, Oren Barkan, Avi Caciu- laru, and Noam Koenigstein. 2021. Self-supervised document similarity ranking via contextualized lan- guage models and hierarchical inference. In Find- ings of the Association for Computational Linguis- tics: ACL-IJCNLP 2021, pages 3088-3098, Online. Association for Computational Linguistics.
Kalpesh Krishna, Aurko Roy, and Mohit lyyer. 2021. Hurdles to progress in long-form question answering. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, pages 4940-4955 Online, Association for Computa- tional Linguistics.
Kartik Goyal, Chris Dyer, and Taylor Berg- Kirkpatrick. 2017. Differentiable scheduled sampling for credit assignment. In Proceedings of the 55th Annual Meet- ing of the Association for Computational Linguistics (Volume 2: Short Papers), pages 366-371, Vancouver. Canada. Association for Computational Linguistics.
Philippe Laban, Andrew Hsi, John Canny, and Marti A. Hearst. 2020. The summary loop: Learning to write abstractive summaries without examples. In Proceed- ings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1135-5150, On- line. Association for Computational Linguistics.
Felix Hill, Kyunghyun Cho, and Anna Korhonen. 2016. Learning distributed representations of sen- tences from unlabelled data. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, pages 1367-1377. San Diego, California, Association for Computational Linguistics.
Md Tahmid Rahman Laskar, Enamul Hoque, and Jimmy Huang. 2020. Query focused abstractive summariza- tion via incorporating query relevance and transfer learning with transformer models. Advances in Artificial Intelligence: 33rd Canadian Conference Artificial Intelligence, Canadian Al 2020, Ottawa, ON, Canada, May 13-15, 2020, Proceedings, page 342-348, Berlin, Heidelberg. Springer- Verrnggr-Veellg..
Mohit Iyyer. Varun Manjunatha, Jordan Boyd-Graber, and Hal Daumé III. 2015. Deep unordered compo- sition rivals syntactic methods for text classification. In Proceedings of the 53rd Annual Meeting of the As- sociation for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing |Volume 1: Long Papers), pages 168 1- 1691, Beijing. China. Association for Computational Linguistics.
Graber,
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sed to-sequence pre-training for natural language generation, translation, and com- prehension. In Proceedings of the 58th Annual Meer- ing of the Association for Computational Linguistics, pages 787 7871-7880, Online, Association for Computa- tional I. Linguistics.
N. Jardine and C.J. van Rijsbergen. 1971. The use of hierarchic clustering in information retrieval. Infor- mation Storage and Retrieval, 755:2177240.
Ziwei Ji. Nayeon Lee, Rita Frieske. Tiezheng Yu. Dan Su, Yan Xu, Etsuko Ishii, Yejin Bang, Andrea Madotto, and Pascale Fung, 2022. Survey of halluci- nation in natural language generation. ACM Comput. Surv. Just Accepted.
Siyao Li, Deren Lei, Pengda Qin, and William Yang Wang. 2019. Deep reinforcement learning with dis- tributional semantic rewards for abstractive summa- rization. In Proceedings of the 2019 Conference on

---

Empirical Methods Natural Language Processing and the 9th International Joint Conference on Natu- ral Language Processing (EMNI,-IIICLL) pages 6038-6044, Hong Kong, China, Association for Com- putational Linguistics.
Romain Paulus, Caiming Xiong, and Richard Socher, 2017. A deep reinforced model for abstractive sum- marization. CoRR, abs//705.044 705.04304.
Fabio Petroni, Aleksandra Piktus, Angela Fan, Patrick Lewis, Majid Yazdani, Nicola De Cao, James Thome,, Yacine Jernite. Vladimir Karpukhin, Jean Maillard. Vassilis Plachouras, Tim Rocktäschel, and Sebastian Riedel. 2021. KILT: a benchmark for knowledge intensive language tasks. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. pages 2523-2544 Online. Association for Computational Linguisties.
Yang Liu and Mirella Lapata. 2019. Text summariza- tion with pretrained encoders. In Proceedings of the 2019 Conference on Empirical Methods in Natu- ral Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3730-3740, Hong Kong, China, Association for Computational Linguistics.
Mary McHugh. 2012. Interrater reliability: The kappa statistic. Biochemia medica: : časopis Hrvatskogu društva medicinskih biokemičara/ / /HDMB, 22:276- 82.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Πlya Sutskever. 2019. Language models are unsupervised multitask learners.
Tsvetomila Mihaylova and André F. T. Martins. 2019. Scheduled sampling for transformers. In Proceed- ings of the 57th Annual Meeting of the Association for Computational Linguisties: Student Research Work- shop, pages 351 1556, Florence, Italy. Association for Computational Linguistics.
Nils Reimers and Iryna Gurevych. 2019. Sentence- BERT: Sentence embeddings using Siamese BERT- networks. In Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natu- ral Language Processing 3982-3992, Hong Kong, China. Association for Com- putational Linguistics.
pages
Diego Mollá and Christopher Jones. 2019. Clas- sification betters regression in query-based multi- document summarisation techniques for question an- swering: Macquarie university bioasq7b. CoRR, abs/1909.00542.
Ruiyang Ren, Shangwen Lv, Yingqi Qu Jing Liu, Wayne Xin Zhao, QiaoQiao She, Hua Wu, Haifeng Wang. and Ji-Rong Wen. 2021. PAIR: Leverag- ing passage-centric similarity relation for improving dense passage retrieval. In Findings of the Associa- rion for Computational Linguistics: ACL-IJCNLP 2021, pages 2173-2183, Online. Association for Computational Linguistics.
Diego Mollá, Christopher Jones, and Vincent Nguyen. 2020. Query focused multi-document summarisation of biomedical texts. CoRR, abs/2008.1 1986.
Ramesh Nallapati, Bowen Zhou, Cicero dos Santos, Çağlar Gulçehre, and Bing Xiang, 2016. Abstrac- tive text summarization using tereenuue----eeeeeeeeeee RNNs and beyond. In Proceedings of the 20th SIGNLL Conference on Computational Natural Lan- guage Learning, pages 280-290, Berlin, Germany. Association for Computational Linguistics.
Alexander M. Rush, Sumit Chopra, and Jason Weston. 2015. A neural attention model for abstractive sen- tence summarization. In Proceedings of the 2015 Conference on Empirical Methods in Natural Lan- guage Processing, pages 379-389, Lisbon, Portugal. Association for Computational Linguistics.
Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018. Don't give the details, just the summary! topic-aware convolutional neural networks for ex- treme summarization. CoRR, abs/1808..r 08745..
Ori Shapira, Ramakanth Pasumuru, Mohit Bansal, Ido Dagan, and Yael Amsterdamer. 2022. Interactive query-assisted summarization via deep reinforcement learning. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies. pages 2551-2568, Scattle, United States. Association for Computational Linguistics.
Preksha Nema, Mitesh M. Khapra, Anirban Laha, and Balaraman Ravindran. 2017. Diversity driven atten- tion model for query-based abstractive summariza- tion. In P Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Pupers), pages 1063-1072, Vancouver, Canada. Association for Computational Linguistics.
David Silver, Satinder Singh, Doina Precup, and Richard S. Sutton. 2021. Reward is enough. Ar- rificial Intelligence, 299:103535:
Ramakanth Pasunuru and Mohit Bansal. 2018. Multi- reward reinforced summarization with saliency and entailment. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume 2 (Short Papers), pages 646,653, New Orleans, Louisiana, Association for Computa- tional Linguistics.
Roberta A. Sinoara, Jose Camacho Coocco-Collados,, Rafael G. Rossi, Roberto Navigli, and Solange O. Rezende. 2019. Knowledge-enhaneed document embeddings for text classification. 163:955-971.
soowledge-Based Systems,
Dan Su, Xiaoguang Li, Jindi Zhang. Lifeng Shang, Xin Jiang, Qun Liu, and Pascale Fung. 2022. Read before

---

generate! faithful long form question answering with
2022) example (absent in the document). We note the absence of such hallucination in the Reinforce- ment Learning (RL) models. However, we note that only of the 5 RL-based models are able to incorporate both aspects of the query. We also note digressions in the RL-based models, with BART R- SEM generating significantly more unrelated con- tent. We observe that this is a general trend: BART R-SEM often generates content very haphazardly. We hypothesize that this is caused by using an ap- proximate semantic similarity scheme as a reward signal.
machine reading. In Findings of the Association for Computational Linguisties: ACI. 2022, pages 744- 756, Dublin, Ireland, Association for Computational Linguistics.
Richard S. Sutton and Andrew G. Barto. 2018. Rein- forcement Learning: An Introduction, second edition. The MIT Press.
Anastasios Tombros and Mark Sanderson. 1998. Ad- vantages of query biased summaries in information retrieval.
Lakshmi Vikraman, Ali Montazeralghaem, Helia Hashemi, W. Bruce Croft, and James Allan. 2021. Passage similarity and diversification non-factoid question answering. In Proceedings of the 2021 ACM SIGIR International Conference on Theory of Infor- mation Retrieval, ICTIR '21, page 27 1-200, New York, NY, USA. Association for Computing Machin- ery.
We also note repeated generations (not neces- sarily consecutive) with same word stems (such as percent, leading to repeated percentage in the BART R R-SIPEG generation) in the RL-based mod- els, trained using ROUGE reward.
Ronald J. Williams and David Zipser. 1989. A learning algorithm for continually running fully recurrent neu- ral networks. Neural Computation, 12):270-280.
B Comparison of Models for Multiple Queries on Same Document
We present two randomly sampled instances, two queries with the same document, from our dataset in Table 19: we include the query and the genera- tions from BART SL and BART R model only, we omit the document for readability.
Yumo Xu and Mirella Lapata. 2021. Generating query focused summaries from query-free resources. In Proceedings of the 59th Annual Meeting of the Asso- ciation for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume I: Long Papers), pages 6096- 6109, Online. Association for Computational Lin- guistics.
For the first query, BART SL hallucinates that Green Revolution has not helped India, given that the document states otherwise. BART R model picks up the key arguments from the document and indeed presents a decent summary pertaining to Green Revolution's help to India.
Yumo Xu and Mirella Lapata. 2022. Document sum- marization with latent queries. Transactions of the Association for Computational Linguistics, 10:623
638.
For the second query, BART SL ignores the query altogether and generates some related sen- tences only. While BART R picks up key infor- mation related to Buffer Stock and articulates satisfactory summary again.
Liu Yang, Mingyang Zhang, Cheng Li, Michael Ben- dersky, and Mare Najork. 2020. Beyond 512 to- kens: Siamese multi-depth transformer-based hierar- chical encoder for form document matching. In Proceedings of the 29th ACM International Confer- ence on Information amp: Knowledge Management, CIKM '20, page 1725-1734, New York, NY, USA. Association for Computing Machinery.
C Comparison of Models for QfS with Random document
A Comparison of Generations from the Models
We present two random samples from our test set where the documents have been replaced by an- other random, unrelated document, in Table 20. We see that for the given query, BART SL and BART R have unrelated (and different) documents. BART SL manages to generate content very much related to the query, which indicates unfair strong in- fluence of the query on the generation, much that the document gets ignored. However, BART generates content from the document indicating that document indeed has appreciable influence on its generation, which is the ideal case in QfS.
We present a randomly sampled instance from our test dataset in Table 18: we include the query and the generations from all six models, we omit the document for readability.
We can see that the BART SL model generates very crisp response to the query, stating what pop- ulation growth is. However, it fails to respond to the other aspect of the query: What does popula- tion change indicate for an area? It also moves on to generate an ill-stated hallucinated (Ji et al.,

---

D Results DebatePedia
gatherable if and only if the post has a score' of atleast 2.
We generate result the test set of DebatePedia using best RL model: BART R-SPPEG. We utilize the model trained on EL15 directly gen- erate results on the test set of DebatePedia. We observe that we perform atleast as good as mod- els trained/fine-tune tuned on DebatePedia, proving the generalization of our approach.
We consider comment (under a gatherable post) if and only if the comment has a score of atleast 2.
After scraping the dataset, we discard posts with associated comments and comments with less than 50 words Finally, we divide the dataset into train and test sets as follows: we accumulate all the posts (and associated comments) from the ex- plainlikeinfive subreddit into the test set, and use the rest for training.
Model
R-1
R-2
R-L
BERTABS(Liu and Lapata,
13.3
2.8
2.8
2019)
BART (Lewis et al., 2020)
21.4
6.3
18.4
LQSUM (Xu and Lapata,
23.5
7.2
20.6
G Architecture Details
2022)
QfS:
We train Senueenceto-Sequenee model generate summaries, given the query and the docu- ment. Our trained model uses BART (Lewis et al.. 2020) as the backbone, to obtain representations of the query and document, and finally generate the summary. Figure 1 illustrates the input to and output from the architecture. Query and document are concatenated and fed to the encoder, and we expect the decoder to generate the summary. We use pretrained BART-large model as the starting point for our QîS model.
BART R-SFPEG
24.2
6.9
20.7
Table Results on DebatePedia test set. R-I:
ROUGE-I, R-2: ROUGE-2, R-I.; ROUGE-L
E Time and Space Complexity Analysis
We present an analysis for the attention layer only, that is the bottleneck for space utilization and time consumption, input length increases. An L-layered decoder utilizes Onn²) time and space for predicting the next token provided the previous n tokens. If the decoding done without Teacher Forcing, that is th token is generated and then passed through the decoder as an input to gener- ate the (n + 1)th token, then the space and time required is:
Table 10 reports statistics on the length of inputs and outputs. Accordingly, we increase the posi- tional embeddings' length of the BART model in our implementation 1568 (approximately equal to Mvalid + 20laid where μ and a denote average and standard deviation of the encoder input length respectively, Table 10). The first 1024 places the new positional embeddings are initialized using the positional embeddings of the pre-trained BART model.
(1 3 + 22 3 .... n²) = (n²)
Passage Embedding: We train encoder-only model generate passage embeddings. Specifi- cally, we fine-tune the BERT-large model (Devlin et al., 2019) to output passage embeddings through the /CLS) token. Figure depicts the dual encoder scheme used to fine-tune the BERT model. We use Equation 4 and to train the model. In addi- tion to the passage similarity task, we also employ Masked Language Modelling (MLM) task during fine-tuning. We observed that the fine-tuned model experienced an increased validation perplexity for MLM, as compared the pretrained BERT-large
On the other hand, if we employ the two pass decoding using Scheduled Sampling, the time and space utilized is (22 n²) = (n²).
F RPEDT Scraping Scheme
We generate RPEDT by scraping posts (queries) and comments (answers) from 39 subreddits. Table 21 presents the list of the subreddits we use to generate the dataset. We gather data within the timeframe: July, 2011 and July, 2022. We set the following criteria to filter the gathered posts and comments:
We compute score as the difference between the upvotes downvotes, score upuotes downvotes "We use SpaCy to tokenize the comments.
1. We consider post (and related comments)

---

Summary
Split
Statistic
Average
Std. Dev.
Enc Inp Len
1120.06
260.14
Train
Dec Out Len
149.07
167.09
BART
Enc Inp Len 1122.85
250.48
Valid
Dec Out Len
147.08
165.55
Enc Inp Len 1123.07
255.31
Test
Dec Out Len
147.24
162.62
Query
Document
Table 10: Statistics the lengths inputs (Ene Inp Len) to the encoder (query + document, concatenated) and outputs (Dee Out Len) from the decoder (sum- mary). The statistics are obtained after tokenization using BART tokenizer.
Figure 1: Architecture for the Query focused Summa- rizer.
Similrrtty
Sigmedd
Hyperparameter
Value
Max Epochs
Learning Rate
2 (SL) /5 7 (RL)
Effective Batch Size
128
REAT
REOT
Max Sequence Length
1568
Optimizer
Adam
Passage1 1
Parsage 2
Table 11: Values of various hyperparameters used while
Figure 2: Dual Encoder schematic for training the Pas- sage Embedding model.
training QSS models. Effective batch size the number of samples passed through the model before parameter updates. SL: Supervised Learning, RL; Reinforvement Learning.
model. We mirror the schemes for MLM train- ing suggested by Devlin al. (2019), during fine- tuning. We keep the default maximum length of BERT (512 tokens), as we see that the average gen- erations from the QfS model are expected to be 150 tokens.
for training. We use the combined loss (Passage Similarity loss, Equation 5, + MLM loss) to train the models. We observe that models fine-tuned for single epoch perform best on the test set of RPEDT. Appendix J presents comparison of the two trained models, in terms of the downstream task performance.
Training Details for QIS
We use the BART-large (12 layers, 16 attention heads, 1024 dimensional embedding) pretrained model as our starting point. We train 8 A100 machines, with all the models taking a total time of 21 days to be trained till convergence. Table outlines the hyperparameters used for training. We train all the models without any explicit learn- ing rate scheduler, and found this to lead to faster convergence in our experiments.
Hyperparameter
Value
Max Epochs
2
Learning Rate
1
Effective Batch Size
1096
Max Sequence Length
512
Optimizer
Adam
I Training Details for Passage Embedding
Values of various hyperparameters used while training Passage Embedding model. Effective batch size is the number of samples (positive labelled instances + in batch sampled negative labelled instances) passed through the model before parameter updates.
Table 12:
We fine-tune both BERT-base and BERT-large (both cased) pretrained models. We train on 2 A 1100 machines, with the base model taking a total 64 hours and the large model taking a total of 186 hours. Table 12 lists the hyperparameters used

---

J Bert-large vs BERT-base for Passage Embedding
process, we noted that multiple questions could be answered from consccutive paragraphs of the chapters. Using this, we copied material relevant to multiple queries from the chapter to form a single document, ensuring absence of topic centralization in RQFT (Baumel et al., 2016). As these were consecutive paragraphs, coherence was trivially en- sured.
We compare BERT-large and BERT-base, as pas- sage embedding models, based on the quality of the models in the downstream task of QfS. We use the fine-tuned passage embedding models reward generated summaries, on how similar they are to the ground truth summaries, while training the Qí' model using Reinforcement Learning. Table 13 presents the results obtained from two QfS models: BART R R-SFPEG-L, (reward: ROUGE + SFPEG from the BERT-large model) and BART R-SFPEG- B (reward: ROUGE SFPEG from the BERT-base model). We observe that BERT-large led signif- icantly better results, and hence proceeded use BERT-large fine-tuned passage embedding model for further experiments, such as human evaluation, in our work.
We construct the dataset from Wikipedia and High School textbooks. The domains from Wikipedia involve: POLITICS, FAMOUS PER- SONALITY, COUNTRY, VIDEO GAME and TV SERIES. The domains from High School textbooks include SCIENCE, SOCIAL SCIENCE, BIOL- OGY, BUSINESS STUDIES, GEOGRAPHY, HIS- TORY, POLITICAL SCIENCE and SOCIOLOGY. Effectively the dataset includes samples from 13 domains.
L Quality Check: RQFT
Metrics
Experiment
Model
We report quality check results performed by two independent annotators. We design the quality check framework as follows:
R-1
R-2
R-L
M-1
39.52
6.92
34.92
EXPT-I
M-II
38.74
6.27
33.50
1.
Each annotator is presented with 30 <query, document, summary> triplets, of which 10 samples (random- (random-samples) are such that the summary does not correspond to the query and document. We keep from random- samples and 10 from the rest 20 (correct-samples) common between the two annotators to com- pute the Cohen Kappa score.
M-I
45.52
16.83
41.29
EXPT-II
M-11
43.94
15.42
40.06
Table 13: Comparison of performance of BART R-
SFPEG-L (M-1) and BART R-SEPEG-B (M-I1) for EXPI-I and EXPr-II.
K Details on RQFT Curation and Domains
2. Each annotator is asked to mark YES (score 1) NO (score 0) to the following questions:
We curate the dataset from two sources- (i) Wikipedia and (ii) high school textbooks. While preparing samples from Wikipedia, we manually copy text from the website, form queries the text data, and write a summary based on the query. High school textbooks contain chapters followed by open-ended questions on concepts within the chapters. We use the questions at the end of the chapter to form our dataset. The answers to the questions are also freely available from online sources, which aid students in their exams. We create <query, document, summary> triplets by (a) Copying material from the chapter (relevant the query) and (b) Copying the answer the question from an online source (the curator verifies whether or not the answer is correct by reading the question and the extracted document). While following this
(a) "Is the query relevant to the document." (ANN-1): This ensures that the dataset contains queries that are synchronous with the paired document. The ideal score is for both random-sample and correct-samples
(b) "Does the summary answer the query while being faithful to the document?" (ANN-2): This ensures that the written summary is indeed correct and is not hallucinated (Ji et al., 2022). The ideal score is for random sandoo--samplss and for correct-samples.
(c)
"Is the summary granmatically correct, fluent, and coherent?" (ANN-3): This ensures the readability, fluency, and

---

grammatical correctness of the written summary. The ideal score is for both rundom-samples and correct-samples.
Table 14 presents the results obtained from the quality check experiment. The values represent the average score the annotators assigned for the annotations: ANN-1, ANN-2, and ANN-3, over the 30 samples. Using the 15 common samples, observe a perfect Cohen Kappa (1.0) between the annotators.
ANN-2
ANN-1
ANN-3
+
Annotator 1
1.0
0.95
0.0
0.95
Annotator 2
1.0
1.0
0.0
1.0
Table 14: Quality scores assigned by annotators and 2 for ANN-1, ANN-2, and ANN-3. correct-samples,
- random-samples.
M Quality Check: RPEDT
RP1
RPz
In order to validate the quality of RPEDT, we pro- vide <Q, P1, P2> triplets to annotators, where Pı and P2 answer the query, Q. We ask them to rate how well does each passage (Pı and P2) answer the query, on scale of 0 (bad) employ two annotators for the task, who rate a total 707 triplets (79 common). Table 16 presents the results from annotation. We measure agreement by noting the number of times they rate passages with the same score. We observe that annotators agree 78.1% on the rating for P1 and 86% on the rating for P2.
Annotator
1.79
1.73
Annotator 2
1.87
1.86
Table 16: Average ratings provided by the two annota-
(good). We
tors for P1 (Rp,) and P2 (Rp.), higher is better.
Score
Criteria
Passage is not related to the topic,
0
does not discuss anything
relevant to the question
Passage partially answers the question
1
Passage completely answers
2
the question
Table 15: Annotation guidelines for the quality check RPEDT.

---

Query: Why do stars twinkle? And, why don't planets twinkle?
document: The twinkling of a star is due to atmospheric refraction of starlight. The starlight, on entering the earth's atmosphere, undergoes refraction continuously before it reaches the earth. The atmospheric refraction occurs in medium of gradually changing refractive index. Since the atmosphere bends starlight towards the normal, the apparent position of the star is slightly different from its actual position. The star appears slightly higher (above) than its actual position when viewed near the horizon. Further, this apparent position of the star is not stationary, but keeps on changing slightly, since the physical conditions the earth's 's atmosphere are not stationary, as was the case in the previous paragraph. Since the stars are very distant, they approximate point-sized sources of light....
Summary: Stars appear to twinkle due to atmospheric refraction. The light of a star after the entry of light into the earth's 's atmosphere undergoes refraction continuously till it reaches the surface of the earth. Stars are far away. So, they are point source of light. As the path of light coming from stars keeps changing, thus the apparent position of stars keeps changing and the amount of light from stars entering the eye keeps twinkling. Due to this, star sometimes appears bright and sometimes dim, which is the effect of twinkling. The planets are much nearer the earth than the stars and because of this, they can be considered as large source of light. a planet considered to be a collection of a very large number point sources of light, then the average value of change in the amount of light entering the eye from all point-size light sources is zero. Due to this the effect of twinkling is nullified.
Table 17: Example of randomly sampled <query, document, summary> triplet from the annotated dataset.

---

Query: What is population growth? What does population change indicate for an area?
BART SL: Population growth refers to the change in number of inhabitants of a territory during
specific period of time. It can be expressed either in absolute numbers or in terms of percentage percentage of the total population For example let 's say you have a region with population of 1,000 people.. Over the next 10 years, there will be increase in population growth as more people move in and out of that area. This means that the population will increase by 10 %
BART R: Population change refers to the change in number of inhabitants of territory during specific period of time. This change may be positive well negative. Population growth occurs not only by increasing births rate but also due to decreasing death rate. There are two ways by which the population size changes - births and deaths. The crude birth rate (CBR) is a simple method of measuring the rate of births in a particular region. can be expressed either in terms of absolute numbers or in numbers of live births per thousand of population. Similarly, the mortality rate is an important indicator of the region's 's demographic structure, social advancement and levels of its economic development.
Population change refers to the change in number of inhabitants of a territory during a specific period of time. This change may be positive as well as negative. Population growth occurs not only by increasing births rate but also due to decreasing death vutc.Population change is an important indicator of economic development, social upliftment and historical and cultural background of the region. It can be expressed either in absolute numbers in terms of percentage. Percentage is the percentage of population in particular geographical area that is increasing or decreasing over time period as a function of its demographic structure, socio-cconomic advancement and other factors such as its geographical location, population density, age, sex ratio, etc. (%) P%)olaalaion growth is a dynamic process that affects the population size over time as it changes from one place to another.
BART R-SFPEG:
BART SFPEG: Population change refers to the change in number of inhabitants of a territory during specific period of time. This change may be positive as well as negative. Population growth occurs not only by increasing births rate but also due to decreasing death rate. populaapln change is an important indicator of economic development, social upliftment and historical and cultural background of the region. It can be expressed either in absolute numbers or in terms of percentage. There are two ways by which the population size changes births, deaths and migration. The crude birth rate (CBR) is expressed as the ratio of live births in year per thousand of population. Conversely, the mortality rate is simple method of measuring mortality of any area.
BART R-SEM: By and large mortality rates are affected by the region's demographic structure, social advancement and levels of its economic development. Population growth occurs not only by increasing births rate but also due to decreasing death rate. When people move from one place to another places, their population may increase or decrease depending on their place of origin and the place they move to is called the Place of destination. Population change refers to the change in the total population of a territory during a specific period of time. Population change an important indicator of economic and social development, socio-economic status and historical and cultural background of the territory.
Population change refers to the change in number of inhabitants of a territory during a specific period of time. This change may be positive as well as negative. Population growth occurs not only by increasing births rate but also due to decreasing death rate. There are different ways by which the population size changes in a region depending on the region's 's demographic structure, social advancement and levels of its economic development. It can be expressed either in absolute numbers or in terms of percentage. The crude birth rate (CBR) is simple method of measuring mortality of any area and it is expressed in numbers of live births per thousand of population for a particular region.
BART R-B:
Table 18: Comparison of generations from all the trained models. The query has two aspects, color-coded by teal and violet. Text-segments in generations are colored accordingly based on which part of the query they answer.

---

Query One: How has green revolution helped India, if at all?
Query Two: Why buffer stock created by the government?
BART SL: I do n't think the green revolution has helped at all The food insecure people are disproportionately large some regions of the country because of poverty tribal and remote areas, regions more prone to natural disasters etc In fact, the states of Uttar Pradesh (eastern and south eastern parts of India), ) Orissa Andaman and Nicobar Islands, and West Bengal have been experiencing famine since the 1970 s
BART SL: If there is shortage of food in region, the government can buy from the market at a lower price than what people are willing to pay for it, and then distribute to the people who need it This way. they do n't have to worry about people going hungry all the time.
BART R: The food insecure people are dispro- portionately large in some regions of the coun- try, such economically backward states with high incidence of poverty, tribal and remote areas, regions more prone to natural disasters etc. Af- ter Independence, Indian policymakers adopted all measures to achieve suff-suffic in food grains. India adopted new strategy in agricul- ture, which resulted in 'Green Revolution', espe- cially in the production of wheat and rice. The farmers are paid pre-announced price for their crops. This price is called Minimum Support Price (MSP). The MSP is declared by the gov- ernment every year before the sowing season to provide incentives to farmers for raising the pro- ductivity of these crops and to encourage them to increase their production in order to reduce their dependence on imported foodgrains (such as wheat, rice, maize, sugar, pulses, oilseeds etc.). In the 196s1 wheat was the most productive crop India and it was grown in a large number of small and medium-scale farms rural areas and in small towns and villages. These small farms were able to produce large quantities of foodgrain and the farmers were rewarded with subsidy of Rs 1,000 per acre per hectare
BART R: Since the advent of the Green Revolu- tion in the carly-1970s, the country has avoided famine even during adverse weather conditions. India aiming at Self-sufficieney in Foodgrains since Independence. The attainment of food secu- rity therefore involves eliminating current hunger and reducing the risks of future hunger. After Independence, Indian policymakers adopted all measures to achieve self-sufficient in food grains. This system has two components: (a) buffer stock. and (b) public distribution system. Buffer Stock is the stocks of grains procured by the government through the Food Corporation of India (FCI). The FCI purchases wheat and rice from the farmers in states where there surplus production. These grains are sold to the consumers at a price lower than the market price of these grains, which is known as the Minimum Support Price (MSP). In the case of wheat rice, it is sold at the MSP of Rs 2,000 per tonne and in case for rice Rs 1,500/tonne of rice the price is Rs 3,200/towards 4 4,300/tons for the same amount grain. There are a buffer price to be paid to farmers for procur- ing the grains at these prices.
system.Buffer
Table 19: Comparison of BART SL and BART R in the case of multiple queries over the same document. Text in red highlights the hallucinated segment the generation.

---

Query: Why do organisms need water?
Document: Primary activities include agriculture, animal husbandry, forestry, fishing, mining and quarrying, etc. Secondary activitics include manufacturing industry, building and construction work, etc. Tertiary activitics include transport, communications, commerce, administration and other services. The proportion of people working in different actívities varies in developed and developing countries. Developed nations have high proportion of people in secondary, and tertiary activities. Developing countries tend to have higher proportion of their workforce engaged in primary activities.
BART SL: Water is an essential part of the life cycle of every living thing. It 's what keeps them alive and allows them to grow and reproduce. Without it they would n't be able to do all the things they do. For example, water is essential for photosynthesis, which is the process of turning food into energy that can be used by other organisms.
During 1998, tropical cyclones formed within seven different tropical cyclone basins, located within various parts of the Atlantic, Pacific, and Indian Oceans. During the year, total of 125 tropical cyclones had formed, with 72 of them have been named by various weather agencies when they attained maximum sustained winds of 35 knots (65 km/h, 40 mph). The strongest tropical cyclones of the year are Zeb, Ron and Susan which peaked with a pressure of 900 hPa (26.58 inHg). Hurricane Mitch late
Document:
BART The most notable storms were Hurricane Georges and Hurricane Mitch. Four Category 5 tropical cyclones were formed 1998. Although a semi-permanent anticyclone persisted through the summer of 1998, causing most of the storms to remain at sea, some storm did threaten the Baja California Peninsula due to a weak ridge of high pressure in the mid-latitudes, and the subtropical low pressure area over the central and castern Pacific Ocean (Pacific Oscillation (POW)). However, the most powerful storms that formed during the season were the hurricanes and tropical depressions (tropical storms). The most destructive storms of this year were hurricanes, which caused significant damage and killed at least 11,000 people, making the system the second deadliest Atlantic hurricane in recorded history, behind only the Great Hurricane of 1780. In the North Atlantic basin, was the deadliest hurricane season on record, with the damages amounting to $9.37 billion and $6.08 billion (1998 USD) in damage, respectively. The scason had above average activity, featuring a record-breaking 30 tropical storms, as well as a high number of hurricanes with a maximum sustained winds of more than 111 mph (178 km/h).
Table 20: Comparison BART R and BART SL. for QîS from random document.
explainlikeimfive
Showerthoughts
NoStupidQuestions
Lightbulb
tipofmytongue
legaladvice
Advice
needadvice
answers
TooAfraidToAsk
AskReddit
askscience
AskCulinary
AskSocialScience
AskEngineers
TrueAskReddit
AskDoes
Crazyldeas
explainlikeIAmA
AskHistorians
askphilosophy
ExplainLikelImaahve
escareerquestions
gamedev
engineering
biology
chemistry
Physics
Fan'T Theories
gardening
writing
Survival
ReverseEngineerring
Hcalth
Cooking
slowcooking
socialskills
datascience
nutrition
Table 21: List of subreddits used for scraping


Reviews:
[{"review_id": "5tyJHmrigC", "reviewer": "Reviewer_VnCj", "summary": "", "questions": "What is the rationale behind building the rewards on ROUGE and evaluating on ROUGE? \n\nIs ROUGE an appropriate evaluation metric for these type of summaries? \n\nWhich implementation and version of ROUGE has been used and what parameters have been given? \n\nAre the results and the reported improvements over other approaches statistically significant? \n\nAre there any plans to publish the research artifacts for the research community? ", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their time and their insightful comments and questions. We have attempted to address the questions to the best of our ability given the time constraints.\n\n> Q1. Building rewards on ROUGE and evaluating on ROUGE is a self-fulfilling prophecy -- rationale. Is ROUGE an appropriate evaluation metric?\n\n-- A1. Inconsistency between Train and Test measurement has been a long-standing challenge in Natural Language Generation $\\rightarrow$ Abstractive Summarization ([[1]](https://openreview.net/pdf?id=HkAClQgA-)), Image Captioning ([[10]](https://arxiv.org/pdf/1612.00563.pdf)), Machine Translation ([[11]](https://openreview.net/pdf?id=HkG7hzyvf)), etc. The survey [[2]](https://arxiv.org/abs/1805.09461) covers this in more detail. All these works stress the requirement of consistency between the training and testing metrics. \nROUGE ([[3]](https://aclanthology.org/W04-1013.pdf)) is a standard metric for evaluation of summaries ([[1]](https://openreview.net/pdf?id=HkAClQgA-), [[3]](https://aclanthology.org/W04-1013.pdf), [[4]](https://aclanthology.org/P17-1098.pdf), [[5]](https://aclanthology.org/K16-1028.pdf)), which highly correlates with the human judgement of the quality of summaries. Query focused Summarization also uses ROUGE as the de-facto metric for evaluation ([[4]](https://aclanthology.org/P17-1098.pdf), [[6]](https://aclanthology.org/2022.tacl-1.36/), [[7]](https://aclanthology.org/2021.naacl-main.393.pdf), [[8]](https://aclanthology.org/2022.findings-acl.61.pdf)). Hence, with the premises that `there should be consistency between train and test measurement`, and that `ROUGE is a proven True North metric for summarization (including query-focused summarization)`, optimizing on ROUGE is a logical conclusion. Thank you for pointing this out, we will include this justification in the camera-ready paper.\n\n> Q2. Which implementation and version of ROUGE has been used and what parameters have been given?\n\n-- A2. We use a ROUGE implementation in Python available publicly -- [https://pypi.org/project/rouge/](https://pypi.org/project/rouge/). We shortlist this particular implementation, as it has been used in the baselines ([[9]](https://aclanthology.org/P19-1346.pdf)), for a fair comparison. We keep the default parameters set by the library, similar to the baselines.\n\n> Q3. Are the results and the reported improvements over other approaches statistically significant?\n\n-- A3. Yes, the results are statistically significant. We report the statistical significance test results for the human evaluation metrics (Lines 491-496) -- which show that our proposed models provide statistically significant improvement over the baselines.\n\n> Q4. Are there any plans to publish the research artifacts for the research community?\n\n-- A4. Yes, we plan to release all artifacts, which include -- datasets, model checkpoints, and code.\n\n> Reproducibility Score 1\n\n-- Response: We report the architectures and all the hyperparameters used to train the model, along with the training infrastructure required -- please refer to sections G, H, I, and J in the Appendix for all the details. In our opinion, the entire project should be reproducible using the specified details.\n\n----\n\n* ### References\n\n[1] A Deep Reinforced Model for Abstractive Summarization (Paulus et al., 2018; In ICLR-2018)\n\n[2] Deep Reinforcement Learning For Sequence to Sequence Models (Keneshloo et al., 2018)\n\n[3] ROUGE: A Package for Automatic Evaluation of Summaries (Lin, 2004; In ACL-2004)\n\n[4] Diversity driven attention model for query-based abstractive summarization (Nema et al., 2017; In ACL-2017)\n\n[5] Abstractive Text Summarization using Sequence-to-sequence RNNs and Beyond (Nallapati et al., 2016; In CoNLL-2016)\n\n[6] Document Summarization with Latent Queries (Xu and Lapata, 2021; In TACL-Volume 10, 2022)\n\n[7] Hurdles to Progress in Long-form Question Answering (Krishna et al., 2021; In NAACL-2021)\n\n[8] Read before Generate! Faithful Long Form Question Answering with Machine Reading (Su et al., 2022; In ACL-2022)\n\n[9] ELI5: Long Form Question Answering (Fan et al., 2019; In ACL-2019)\n\n[10] Self-critical Sequence Training for Image Captioning (Rennie et al., 2016; In CVPR-2017)\n\n[11] A differentiable BLEU loss. Analysis and first results (Casas et al., 2018; In ICLR-2018)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The pipeline, the dataset and the comparison would be valuable to the community, if the research artifacts are release. ", "reasons_to_reject": "Building rewards based on ROUGE and evaluating based on ROUGE seems to be a kind of self-fulfilling prophecy.  ", "questions_to_authors": "What is the rationale behind building the rewards on ROUGE and evaluating on ROUGE? \n\nIs ROUGE an appropriate evaluation metric for these type of summaries? \n\nWhich implementation and version of ROUGE has been used and what parameters have been given? \n\nAre the results and the reported improvements over other approaches statistically significant? \n\nAre there any plans to publish the research artifacts for the research community? ", "paper_topic_and_main_contributions": "The paper \"Reinforcement Replaces Supervision: Query focused Summarization using Deep Reinforcement Learning\"  presents work on query focused summarization based on Reinforcement Learning, rather than Supervised Learning. In order to demonstrate the quality of the work, the paper presents a new dataset, human and automatic evaluation and the respective pipeline to achieve the results, which are also compared to two other datasets. ", "excitement": 3, "reproducibility": 1, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "l. 128 \"inspite\"\nl. 226 \"token-by-token match We take the RL...\" missing full stop\nl 520 \"On the other hand....\" -- First hand is missing\n"}, "extra_text": {}}, {"review_id": "X22NGT2iZV", "reviewer": "Reviewer_G1x8", "summary": "", "questions": "- Line 15: Be specific on which datasets.\n", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their time and their insightful comments and questions. We have attempted to address the questions to the best of our ability given the time constraints.\n\n> Q1. QMSum, SQuALITY, and SOCRATIC Pretraining\n\n-- A1. We do not report results on QMSum and SQuALITY primarily for the following reason: \n* The document lengths, $9070$ and $5200$ respectively, exceed our models' maximum length limit ($1568$) by a huge margin. For reasons of time and space constraints of training our model, we select a dataset with decently sized documents ($1120$ tokens, average) and summaries ($150$ tokens, average).\n\nAdditionally, the queries (questions) in the QMSum dataset come from a limited set of $13$ templates. In contrast to this, the queries in ELI5, DebatePedia, and our proposed test set are quite diverse. The significant improvements on these datasets thus validate that our proposed models generalize quite well.\n\nWhile we acknowledge the results of the recent (ACL 2023) paper on SOCRATIC pretraining, a fair comparison of RL-based methods for such long inputs would require significant changes to the approach (such as including the Segment Encoder, [[1]](https://aclanthology.org/2022.findings-naacl.109.pdf), to reduce resource requirement) -- indicating that `this is better addressed in a future work, rather than the presented work`.\n\n> Q2. The authors might want to double-check whether the claim in line 84 is correct. Check \"Teacher Forcing Recovers Reward Functions for Text Generation (Hao et al., 2022)\"\n\n-- A2. The cited work (and other works it cites) is totally orthogonal to our claim in line 84. Our claim is `we are the first work to resolve the conflict of employing RL in Transformers with Teacher Forcing`. Why is Teacher Forcing bad for application of RL to Natural Language Generation? $\\rightarrow$ `Using ground truth tokens implies we are doing no exploration whatsoever, since we stay very close to the reference data set. The point of RL is to let the algorithm explore generations which are not exactly (token to token) the same as the ground truth, but are semantically similar.`\nThe cited work discusses something orthogonal $\\rightarrow$ It proves (theoretically) that a model trained using Teacher Forcing learns an Inverse Reinforcement Learning (IRL) model $\\rightarrow$ basically a reward function -- `this in no way aids or contradicts our claim, nor does it address our claim`.\n\n> Q3. Missing References\n\n-- A3.\n\n* QMSum (Zhong et al., 2021) -- We omit this for reasons in A1.\n* Socratic Pretraining (Pagnoni et al., 2023) -- As addressed in A1, it is not a fair comparison with our proposed models. However, we acknowledge that this would work be a really strong comparison for future work, extending our approach to longer sequences. We thank the reviewer for this suggestion.\n* (Shapira et al, 2022) -- We omit this as it employs RL for **extractive** QfS, and we strictly focus on RL for **abstractive** QfS. However, thank you for the suggestion, and we will be including our literature survey on RL for extractive QfS in our camera-ready paper.\n* (Vig et al., 2022) -- Thank you for suggesting this, we will include it in the camera-ready paper.\n* (Xiong et al., 2022) -- This work presents a new LLM, and would have been related, had we included the QMSum benchmark. However, for the reasons stated above, we omit QMSum, which is why we do not include this work in our literature survey.\n\n----\n\n* ### References\n\n[1] Exploring Neural Models for Query-Focused Summarization (Vig et al., 2022; In NAACL-2022)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The curated datasets (RPEDT for passage embedding models and RQFT for analyses of QFS models) could be useful as a testbed for future research.\n\n- The interesting discovery that RL aids in the acquisition of QFS by the model, even while considering the ROUGE score. Also, the Cluster Hypothesis outperforms rival passage embedders in providing enhanced semantic feedback.", "reasons_to_reject": "- The most common QFS evaluation datasets are QMSum and SQuALITY. I would like to see the comparison w/ SL and w/ RL there, and compare to those existing results reported, for example, BART-LS, BART-Large SegEnc, and BART-Large SegEnc + SOCRATIC Pretraining. \n\n- The authors might want to double-check whether the claim in line 84 is correct. For example, \"Teacher Forcing Recovers Reward Functions for Text Generation (Hao et al., 2022)\" and some of the papers in their related work might be a good starting point to check more details. I am not very familiar with that line of research.", "questions_to_authors": "- Line 15: Be specific on which datasets.\n", "paper_topic_and_main_contributions": "The paper is about Reinforcement Learning based models for the task of QFS. They use multiple rewards with Policy Gradient networks, using signals such as ROUGE, BLEU, and Semantic Similarity,  ", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "- QMSum: A New Benchmark for Query-based Multi-domain Meeting Summarization (Zhong et al., 2021)\n- SOCRATIC Pretraining: Question-Driven Pretraining for Controllable Summarization (Pagnoni et al., 2023)\n- Interactive Query-Assisted Summarization via Deep Reinforcement Learning (Shapira et al, 2022)\n- Exploring Neural Models for Query-Focused Summarization (Vig et al., 2022)\n- Adapting pretrained text-to-text models for long text sequences. (Xiong et al., 2022)\n"}}, {"review_id": "GrbS7Adtwl", "reviewer": "Reviewer_ydP6", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their time and their insightful comments and questions. We have attempted to address the questions to the best of our ability given the time constraints.\n\n> Q1. Recent baselines and SOTA techniques.\n\n-- A1. We present our results on two standard benchmarks -- ELI5 (**Table 6**) and DebatePedia (**Table 9**). DebatePedia has recent baselines, hence we report them and it can be seen we improve over the baselines with our method, in **zero shot setting too** (model trained on ELI5 train set and tested on DebatePedia) -- proving our approaches' superiority the over SOTA techniques for QfS. As for ELI5, recent papers use the KILT version of the dataset, which poses the query-summary pairs in an Information Retrieval format, with no gold support document to summarize from. We use the most recent available baselines. Please refer to **Lines 132-136** and **Lines 415-420** in the paper for justification.\n\n> Q2. Table 6 experimental results not shown for baselines.\n\n-- A2. The baselines do not release their trained models, which is why results on the new dataset could not be obtained. However, we retrain a baseline model (replicating the one by Lewis et al., 2020) -- BART SL, and produce the results in **Table 6**, specifically **Row 3**.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. A new RL-based method that solves the conflict of RL in Transformers with Teacher Forcing.\n2. A new passage embedding-based reward mechanism using the Cluster Hypothesis.\n3. Two new datasets for training and analysing models. ", "reasons_to_reject": "1. This paper only considers two baselines from 2019 and 2020. In a sense, these baselines may already be outdated and not keeping up with the current STOA techniques. I'm not an expert on QfS, but I think newer baselines in the recent 2 years should be considered. \n2. As shown by Table 6, the authors do not report the experimental results of the two baselines on the new datasets created by the authors.\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper focuses on Query-focused Summarization (QfS). This paper proposes RL-based models for QFS. In detail, this paper 1) uses Scheduled Sampling to resolve the conflict of employing RL in Transformers with Teacher Forcing, 2) presents a novel reward derived using the Cluster Hypothesis, 3) contributes the RPEDT dataset to train passage embedding models, and 4) contributes the RQFT dataset for analyses of QfS methods. The extensive experimental results show the effectiveness of the proposed methods.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper introduces Reinforcement Learning (RL) based models for Query-focused Summarization (QFS) inspired by RL in natural language generation. The proposed work employs rewards through ROUGE, BLEU, and Semantic Similarity. It addresses conflicts with Teacher Forcing through Scheduled Sampling. Additionally, the paper introduces a novel reward based on the Cluster Hypothesis, contributes the RPEDT dataset for passage embedding model training, and presents the RQFT dataset for QfS method analysis. The experimental results demonstrate the effectiveness of the proposed method in QFS. Please add baseline results on the new dataset for Fan et al. (2019).
Decision: Accept-Main