Can Large Language Models Fix Data Annotation Errors? An Empirical Study Using Debatepedia for Query-Focused Text Summarization
Authors: Md Tahmid Rahman Laskar
Mizanur Rahman
Israt Jahan
Enamul Hoque
Jimmy Huang
Venue: EMNLP 2023 Findings (2023)
Keywords: ChatGPT
PaLM
Large Language Models
Query Focused Abstractive Text Summarization
TL;DR: Investigated whether Large Language Models can Fix Data Annotation Errors on the Debatepedia dataset to make it suitable for Query-Focused Text Summarization.

Abstract: Debatepedia is a publicly available dataset consisting of arguments and counter-arguments on controversial topics that has been widely used for the single-document query-focused abstractive summarization task in recent years. However, it has been recently found that this dataset is limited by noise and even most queries in this dataset do not have any relevance to the respective document. In this paper, we study whether large language models (LLMs) can be utilized to clean the Debatepedia dataset to make it suitable for query-focused abstractive summarization. More specifically, we harness the language generation capabilities of two LLMs, namely, ChatGPT and PaLM to regenerate its queries. Based on our experiments, we find that solely depending on large language models for query correction may not be very useful for data cleaning. However, we observe that leveraging a rule-based approach for
data sampling followed by query regeneration using LLMs (especially ChatGPT) for the sampled instances may ensure a higher quality version of this dataset suitable for the development of more generalized query-focused text summarization models.

Can Large Language Models Fix Data Annotation Errors? An Empirical Study Using Debatepedia for Query-Focused Text Summarization
Md Tahmid Rahman Laskar Mizanur Rahman Israt Jahan',
Enamul Hoque', Jimmy Xiangji Huang**
'Dialpad Canada Inc., &Royal Bank of Canada, York University! Toronto, Ontario, Canada
(tahmid20, 101200,izzz mizunurr, israt18, 8, enamulh, jhuang)@@orku...
Abstract
One widely used dataset for this task is the De- batepedia dataset that consists of arguments and counter-arguments on conversational topics (Nema et al., 2017). The query-focused summarization of argumentative text is a challenging task that has gained increasing attention in recent years due to its potential applications in various domains, such as policy-making journalism, and legal reasoning. However, it has been recently found that the quality of the Debatepedia dataset that is widely used for this task limited by noise, with many of the queries in this dataset having no relevance with the source document (Laskar et al., 2022). Since Debatepedia is a rich source of argumen- tative text on controversial topics that can serve as valuable resource for the development and evaluation of summarization models, in this paper, we present a novel methodology to clean the De- batepedia dataset via re-annotation of its queries to make it usefuu resource for query-focused ab- stractive summarization. Our data annotation ap- proach leverages large pre-trained language models (Devlin et al., 2018; Brown et al., 2020; Ouyang et al., 2022), such ChatGPT (OpenAl, 2023) and PaLM-2 (Anil al., 2023), that have demon- strated impressive capability of generating fluent and coherent text (Laskar al., 2023a). Using these LLMs, we regenerate the queries in the De- batepedia dataset that are more likely to have no relevance to the document and the summary. More specifically, this paper aims to investigate whether LLMs can be utilized to fix the existing issues in the Debatepedia dataset. Our extensive experiments show that utilizing rule-based filtering to eradicate noisy instances alongside leveraging the generative power of LLMs to regenerate the irrelevant queries leads to performance improvement terms of both query relevance and summary generation quality. We will make this LLM-annotated cleaned version of Debatepedia publicly available",
Debatepedia is a publicly available dataset con- sisting of arguments and counter-arguments on controversial topics that has been widely used for the single-dneument query-focused abstrac- tive summarization task in recent years. How- ever, it has been recently found that this dataset is limited by noise and even most queries in this dataset do not have any relevance to the respec- tive document. To this end, this paper aims to study whether large language models (L.LMs) can be utilized to clean the Debatepedia dataset to make it suitable for query-focused abstrac- tive summarization. More specifically, har- ness the language generation capabilitics of two LLMs, namely, ChatGPT), and PaLM2 to re- generate its queries. Based we find that only fixing the queries in Debate- pedia via LLMs may not be useful. However, leveraging rule-based approach via filtering out noisy instances followed by query regenera- tion using LLMs for the sampled instances may ensure higher quality version of this dataset suitable for the development more general- ized query-focusee text summarization models.
experiments,
1 Introduction
Text summarization a natural language process- ing technique that involves generating a concise and coherent summary of longer piece of text while preserving its most important information (Yao et al., 2017). Query-focused text summariza- tion is specific type of summarization that gener- ates summary of the given text that is focused on answering a specific question (Laskar et al., 2020c) or addressing a particular topie, rather than provid- ing a general overview of the text. (Baumel et al., 2018; Goodwin et al., 2020; Su et al., 2020; Xu and Lapata, 2021; Laskar et al., 2020a.b, 2022).
"Contact Author.
'All work being done at York University. 'https: //openai.ppap////////ee http:///aaaaapnoolpeeee/eeee/e/e hon/blog/chatgpt//
Phttps: //ai htt.s///////ooglll/oeeeeooo /isgg///e///o/caam..
Phttps: / thub. com/tahhmedge/CQsuuup

---

2 Related Work
3 Our Annotation Methodology
Debatepedia publicly available dataset of ar- guments and counter-arguments on debate topics, proposed by Nema et al. (Nema et al., 2017). It con- tains about 13K query-document-suumaa pairs. The average number of words per document, sum- mary, and query in the Debatepedia dataset is 66.4, 11.16, and 9.97, respectively. The dataset covers wide range of topics, such as politics, sports, and technology, and has been extensively used in recent years to build query-based summarization models for argumentative text (Laskar et al., 2022). How- ever, the quality of Debatepedia as a dataset for query-based summarization has lots of limitations (see Table 5 in Appendix A. for some examples), as it has been found recently that many queries in this dataset (Laskar et al.. 2022). To address these limitations. we propose a methodology for cleaning the Dehate- pedia dataset via leveraging two popular LLMs: ChatGPT (Laskar et al., 2023a) and PaLM-2 (Anil et al., 2023), as annotators. In this regard, we ini- tially explored various techniques to identify how to effectively sample the noisy instances, and subse- quently, we regenerated the queries for the sampled instances. We denote ChatGPT and PaLM an- notated versions of Debatepedia (DP) for Query Focused Abstractive Summarization as the CQ- SumDP and the PQSumDP, respectively.
Query-focused text summarization using neural models has gained increasing attention in recent years (Baumel et al., 2018: Laskar et al., 2022). The recent success of transformer-based models (Vaswani et al., 2017; Lewis et al., 2019; Raffel al., 2019; Zhang et al., 2019) on generic abstrac- tive summarization has also inspired researchers to utilize such models for query-based abstractive summarization (Goodwin et al., 2020; Vig et al., 2021; Laskar et al., b020a,b, 2022), leading to state- of-the-art performance in benchmark query-based summarization datasets, such as DUC (Feigenblat et al., 2017; Xu and Lapata, 2020), AQuaMuSe (Kulkarni et al., 2020), QMSum (Zhong et al., 2021), Debatepedia (Nema et al., 2017), etc.
not relevant the document
Among the datasets mentioned above, one no- table exception is the Debatepedia dataset since it requires generating summaries from a docu- ment containing argumentative text (i.e., arguments and counler-arguments)) However, it has been found recently that many samples in the Debate- pedia dataset are not actually query oriented while models that are trained without considering the query relevance could achieve almost similar per- formance as the query- query-focused summarization mod- els (Laskar et al., 2022). Thus, there remains a scarcity of datasets specifically tailored to generate query-focused summaries of argumentative texts.
3.1 Data Sampling
Though some studies (Abdullah and Chali, 2020) have attempted to generate the queries in generic summarization datasets (c.g., CNNDM (Nallapati et al., 2016)), we find that these queries are gen- erated by directly extracting words from the ref- erence summaries, leading to unexpected access to the keywords in the reference summaries for the summarization models. LLMs have received lot of attention recently due their impressive language generation capability - ensuring high flu- ency, coherence, and grammatical correctness on the generated texts (Laskar et al., 2023a; Qin et al., 2023; Bang et al., 2023; Yang et al., 2023; Wang et al., 2023; Kocoń et al., 2023). More importantly, ChatGPT like LLMs also demonstrated impressive capability for data annotation (Wang et al., 2021; Ding et al., 2022; Gilardi et al., 2023). To this end, in this paper, we study how to fix the queries in Debatepedia using L.LMs to construct a cleaned version of the dataset to make it suitable for query- focused summarization of argumentative texts.
We explore two approaches for data sampling. one approach, we study whether only fixing the queries the Debatepedia dataset via leveraging LLMs for query regeneration could address the is- sues in the Debatepedia dataset or not. For this purpose, we ask LLMs to identify the instances in the Debatepedia dataset where the queries seemed irrelevant. In our other approach, we first sam- ple data instances based on some filtering rules by excluding instances that are less relevant for query- focused summarization, and then we ask LLMs to re-generate the queries from these sampled in- stances where the queries looked irrelevant. Our prompt for data sampling using L.MMs is shown in Table (()). Below, we describe these approaches.
(i) LLM-based Data Sampling without Filter- ing: In this approach, we use the full Debatepedia dataset to find the irrelevant queries using L.Mss. For this purpose, we provide each instance of De- batepedia to the LLMs to determine if the query is

---

(a) Prompt: Data Sampling for Query Regeneration
(b) Prompt: Regenerating Sampled Queries
Below; provide query, document, and the focuued summary of the given document. Identify whether the query is relevant the summary? Answer either yes or Query: [QUERY]
A document along with its sunumary given below. Write down the most reasonable query relevant this document-
pair?
Document: [DOCUMENT]
Decument: [DOCUMENT]
Summary: [SUMMARY]
Summary: (SUMMARY)
Table I: Prompts for LLMs: (a) data sampling for query regeneration, and (b) regenerating the sampled queries.
relevant to the document/summary. However, we find a significant difference between LLMs in this task. While PaLM-2 only identifies 659 queries irrelevant (612/19/28 in train/valid/test sets, re- spectively), ChatGPT identifies 6435 queries as irrelevant (5697/31 6/422 in train/valid/test sets, re- spectively), out of 13719 samples.
the training, validation, and test sets, respectively. Below, demonstrate how we utilize LLMs for query regeneration.
3.2 Using LLM for Query Regeneration
We concatenate the document and the reference summary together and give as input to the LLMs for query regeneration. Our sample prompt for this task can be found in Table l(b). While we could ask LLMs to generate both the query and the query- based summary by only giving the document in the input prompt, did not do since it is found that LLMs like ChatGPT tend to generate longer summaries (Laskar et al., 2023a; Qin et al., 2023) while the resulting dataset could become a fully synthetic dataset. Thus, we use both the document and the summary as input and only regenerate the queries while keeping the original reference sum- maries intact. We find that the regenerated queries using ChatGPT and PaLM-2 only have 15.2% and 11.4% word overlaps, respectively, with the gold summaries, in comparison to the 10.6% word over- laps in the original Debatepedia dataset.
(ii) LLM-based Data Sampling via Filtering: In this approach, instead of cleaning the Debatepe- dia dataset by only fixing the queries, we also fol- low some rules to first filter out some irrelevant in- stances from the dataset to address the existing lim- itations in Debatepedia (Laskar et al., 2022), such as smaller-sized documents, close-ended questions, etc. Since for the smaller-sized documents, the ref- erence summaries are mainly the overall generic summary of the document where the additional query does not help, aim to exclude smaller- sized documents ensure that the reference sum- maries are more query-focusee This also helps us to address the noisy scenario the dataset when the reference summary length is longer than the document length. Based on manual analysis, we find that a minimum length of 75 words for each se- lected document at least ensures document where the query could play role in the summary genera- tion. To also address the issue of short summaries that looked like answers closed-ended ques- tions, we exclude instances where the length of the summary is shorter than 5 words. This helps us to clean the dataset in a way such that instead of hav- ing a dataset with close-ended questions and short answers, we propose dataset consisting of concise but coherent summaries. This results in filtered version of the dataset which is smaller in size, con- sisting of 5291/309/405 instances, in comparison to the original d containing 12000/719/100 in- stances, in train/valid/test sets, respectively. We also find that ChatGPT and PaLM-2 identified 2171//20/145 and 218/6/6 queries as irrelevant in
Experimental Results
In this section, we present our experimental find- ings. We denote the version of our dataset where we did not apply any filtering as the unfiltered version, whereas we denote the version of our dataset where we also utilize filtering while sam- pling data instances based on applying some rules as the filtered version. For ChatGPT, we use the 5it-33.-tuurbopppe 0301 model; while for PaLM- 2, we use the b model. We fine- tune the following models benchmark the per- formance in our re-annotated versions of Debatepe- dia since these models achieved impressive perfor- mance in query-focused abstractive summarization
'nttps://plaffomm.oo wow.ppnnai.comm///s//eeeeeesss gpt-3-5
chttps://gitee http://tttttbb.mmm//eressaeae/// divestiilssattaannmmmmccccssreee
"https://clonn httt:s::/cllod.goop/e wow./vertex-ai/ooctttt.. generative- peeratiiveaa/llean//mdellll

---

Modd
Training
Exaluation
ROUGE:1
ROUGE-2
ROUGEL
BERTScore
BART-BaL Base
Original Debutapedia
MS-MARCO
57.8
20.0
34.2
68.8
Pepasis Base
Onginal Dehatopedia
MS MARCO
31.2
14,9
27.3
59.
T5- 15-Base
Original Dehatepedia
MS-MARCO
45.1
26,9
40.9
71.9
Base
CQSumDD (illenel
MS-MARCO
42.3
24.2
38.2
70.5
Pepasis Hase
DQSumby (ffiered)
MS MARCO
41.1
51.4
45.3
69.0
15 Base
CQSunDP Ulllare)
MS-MARCO
47.6
29.4
43.0
73.1
BaRt
PQSumhPP (finena))
MS-MARCO
41.4
23.3
37.5
70.6
Pegasis Base
PQSumUPP (iloered)
MS MARCO
44.7
25.2
40.7
66.7
TS-Base
PQ5amDP (flimaa))
MS-MARCO
47.3
26.0
42.8
73.0
BART-Bass
CQ6mmDP (nniltaaa)
MS-MARCO
38.4
20.6
34,8
68.4
Pegaiss Base
CQUuDDP (unditered)
MS-MARCO
43.7
27.3
40.0
67.7
TS-Rane
CQSumDP (unfiltenal
MS-MARCO
44.7
26.5
40.4
71.9
BAKE Base
IQSumbi (uniitered)
MS MARCO
39.1
21.4
35.5
69.1
Pegaits- Base
PQSauDDP cantitered)
MS MARCO
40.1
23.1
36.3
66.2
TS-Base
PQSumDDP (unfiteere))
MS-MARCO
44.4
25.6
40.0
72.3
Table 2: Performance of differeat models on MS AS-MARCO when trained respective versions of Debatepedia (DP).
ROUGE-1
ROUGE-L
BART-Bae
CQSumDD (ileend))
39.6
22.1
36.6
70.8
COSunDP (illered)
TS-Base
CQ (illened)
BAKI Base
PQSumDP (tinere))
37.8
35.6
70.3
POSumDPP (fttered)
Ta-e
(6hund))
BART Base
CQSumDP (undinered)
41.6
23.4
39.1
72.3
Pegasss-Bae
CQSumDP (unfiteond)
33.4
15.8
30.6
68.3
15-Base
CQsumDP (undincred)
BAKI Base
PUSumDY (antiltere))
39.8
22.2
372
71.7
Pegase- Bose
PQSumDD (untitured)
29.2
26.6
66.3
B3se
POSumDD (uatitteed)
Table 3: Performance of different models on various versions of Dehatepedia.
in recent years (Laskar et al., 2022; Goodwin et al., 2020): (i) BART (Lewis al., 2019), (ii) T5 (Raf- fel et al., 2019), and (iii) Pegasus (Zhang al., 2019). Similar to prior work on query-focused text summarization (Laskar et al., 2022), we concate- nate the query with the document and give as input to these models to generate the query-focused sum- maries. For all models, we use their respective Base versions from HuggingFace (Wolf et al., 2019) (see Appendix A.3 for more details). For all models, the results are evaluated using ROUGE-I, ROUGE-2, ROUGE-L, and BERTScore. For BERTScore, we use the DeBERTa- -alalTTT--larg--mit (He et al.) model.
considering the gold passage the source docu- ment, and after combining the passage with the query we give the concatenated text as input to the models. The results all three models (BART, TS, Pegasus) on MS-MARCO that are fine- sinettnned on the respective versions of Debatepedia are shown in Table 2. We observe based on our experimental results that the domain generalization performance is much better when the CQSumDP/PQSumDD ver- sions of the Debatepedia dataset used in compar- ison using the Original Debatepedia dataset. While comparing ChatGPT and PaLM as data annotators, we observe that models trained on CQSumDP per- form better than PQSumDP. Moreover, we find that models trained on the filtered version obtain bet- ter performance (with T5-Base achieving the best result), indicating the importance of cleaning the Debatepedia dataset by excluding noisy instances, alongside utilizing LLM-generated queries.
4.1 Effectiveness of LLMs for Data Cleaning
this section, to investigate the effectiveness of using LLMss for data cleaning, we evaluate the performance of models trained on different LLM- annotated versions of the Debatepedia dataset in an out-of-domain dataset for the query-focused ab- stractive summarization task. This is done to ensure that all models are evaluated on the same evaluation set. In this regard, we use the development set of the QA-NLG version of the MS-MARCO (Wang et al., 2018) dataset (12467 samples). We follow the similar settings of Laskar et al. (2022) by only
Qualitative Evaluation of Model Generated Summaries: We sample 10 summaries generated by each model (BART, T5, Pegasus) the MS- MARCO dataset conduct human evaluations for our best-performing approach, the CQSumDP (fil- tered version), and the baseline Original Debate- pedia. In our human evaluation, we ask humans

---

score between 1 to 5 for the factual consistency and the coherence of the summaries generated by different models for the given queries. The average coherence and factual consistency scores for mod- els trained on CQ (filtered) are 3.4 and 3.3, respectively; in comparison to the average coher- ence and factual consistency scores of 3 and 2.6. respectively, for the Original Debatepedia. This further establishes the effectiveness of using LLMs as annotators to make more suitable dataset for query-foeusee text summarization.
Type
ROUGE-1
ROUGE-2
ROUGE-L
BERTSoore
ChatGPT Generated Query
47.6
29.4
43.0
73.1
Withoss Query Relevance
70.3
Origiaal Query
29.2
Table 4: Debatepedia (filtered) and evaluated MS-MARCO.
Ablation test result for the T5 Base model fine-tuned tuned
Cost and Time Efficiency: Recently, it was found that LLMs could significantly reduce the labeling cost without sacrificing the model's per- formance much, making it possible to train models on larger datasets without the need for human la- beling (Wang et al., 2021; Ding et al., 2022; Liu et al., 2023). In this work, observe that Chat- GPT/PaLM APIs could generate about 15 queries on average per minute, which should be much faster than using human annotators, since humans may need some time to come up with the most effective query for the given document-summary pairs. This makes LLMs to be more effective for annotation.
Qualitative Evaluation of LLM Generated Queries: We sample 100 instances and ask three human evaluators choose between the ChatGPT and -ggnerreee (see Appendix A.4 for some examples) queries that they prefer based on the conciseness and the relevancy of the query. We find that in 66% cases (via majority voting), the ChatGPT-generateed queries were preferred.
LLM as Query Relevancy Classifier: To mea- sure the capability of LLMs in classifying whether the query is relevant to the document/summarry we sample 100 instances and evaluate using three hu- man evaluators to find if they also agree with the classification done by LLMs. We find based on majority voting that the precision for the classifica- tion task for PaLM-2 is 75,, while for ChatGPT is 63%. This trend of PaLM-2 outperforming Chat- GPT in discriminative tasks (e.g., classification) while being underperformed in generative tasks is also observed in recent studics (Jahan et al., 2023).
4.2 Performance Benchmarking Different Versions of Debatepedia
In this section, we benchmark the performance of various LLM-annotated versions of the Debatepe- dia dataset. We present our results in Table 3 to find that all three models perform better in the CQ- SumDP dataset in comparison to their performance on the PQSumDP. This gives further indication that the queries generated by ChatGPT are more help- ful in improving the model performance. While comparing between different models, we find that in both the filtered and the unfiltered versions, the best performance is achieved by the BART model.
Ablation Studies: To further investigate the use- fulness of L.MM-generaeed queries, conduct the following ablation tests using the best- persfreeronnng model, T5-base, on MS-MARCO (see Table 4).
Conclusions and Future Work
(i) Remove LLM-generated Query: Here, we evaluate the performance of the T5 model by fine- tuning it the filtered version of Debatepedia without incorporating any query relevance. We find based on the average score across different metries that the performance for T5 dropped by 9.53% on average, in comparison to the T5 model fine-tuned on the CQSumDP (filtered) dataset. (ii) Replace LLM- -eM-generated Query: Here, evaluate the performance by fine-tuning T5 using the original query instead of the LLM--genrrted query the filtered version of Debatepedia. Based on the average scores achieved by the T5 model, the performance is dropped by 3.57% on average, compared to T5 fine-tuned on CQSumDP (filtered).
In this paper, we study how to effectively leverage LLMs to construct cleaned version of the Debate- pedia dataset to address the existing limitations in this dataset in order make it suitable for query- focused text summarization. Based on extensive ex- periments and evaluation, we demonstrate that our proposed data re-annotation approach using LLMs (especially ChatGPT) results in cleaner version of Debatepedia that is found to be more effective for the focused summarization task in compari- son to the original dataset. In the future, we will ex- plore whether few-shot examples with LLMs lead to better performance. Our re-annotated versions of Debatepedia will also be made publicly available here: https: / //gi ligitiin thub. com/ tahnedge/CQquuMPP.

---

6 Limitations
References
ChatGPT (GPT-3.5) and PaLM models are contin- uously upgraded by OpenAl and Google. Thus, it may not be possible to reproduce the same queries using these models. However, this also mimics the real-world scenario as different human annota- tors may write different queries (e.g., in many text summarization datasets, there can be multiple gold reference summaries written by different human annotators). However, similar to the work of Guo et al. (2023), we also notice that this difference is very small. Therefore, we also generate only one query for each example. Though new version of ChatGPT called GPT-4" has been recently released which may generate more powerful queries, in this work, we did not utilize GPT-4 as it is quite expen- sive to use than the original ChatGPT (i.e., GPT- 3.5) while being significantly slower. Nonetheless, future work may compare with other more power- ful LLMs (including GPT-4) for data annotation.
Deen Mohammad Abdullah and Yllias Chali. 2020. To- wards generating query to perform query focused abstractive summarization using pre- pre-trained model. In Proceedings of the 13th International Conference Natural Language Generation, pages 80-85.
Rohan Anil, Andrew M Dai, Orhan Firat, Melvin John- son, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuell Taropa, Paige Bailey, Zhifeng Chen, al. 2023. Palm 2 technical report. arXiv preprint arXiv:2305. 200.
Yejin Bang, Samuel Cahyawijaya, Naycon Lee, Wen- liang Dai, Dan Su, Bryan Wilie. Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, et al. 2023. A multi- task, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity. arXiv preprint arXiv:2302.040: 04023.....
Tal Baumel, Matan Eyal, and Michael Elhadad. 2018. Query focused abstractive summarization: Incorpo- rating query relevance, multi-document coverage, and summary length constraints into seq2seq models. arXiv preprint PrxXiv:800.000 1801.07704.
Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neclakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. arXiv preprint arXiv:2005. 14165.
7 Ethics Statement
Since this paper only utilizes LLMs generate the queries for the given sucumrrn-summary pairs, it does not lead to any unwanted biases or ethical concerns. However, all the responses generated by ChatGPT and PaLM are still manually checked by the authors to ensure that the LLM-generated queries in the cleaned version of the dataset do not pose any ethical concerns or unwanted biases. Only used that
Jacob Devlin, Ming-Was Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understand- ing. Proceedings of the Annual Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technolo- gies, 4171-4186.
publicly available academic dataset did not require any licensing. Thus, no personally identifiable information has been used while uti- lizing LLMs to fix the queries the Debatepedia dataset. All the human evaluators were also paid above the minimum wage.
Bosheng Ding, Chengwei Qin, Linlin Liu, Lidong Bing. Shafiq Joty, and Boyang Li. 2022. Is gpt-3 a good data annotator? arXiv preprint arXiv:2212.
Guy Feigenblat, Haggai Roitman, Odellia Boni, and David Konopnicki. 2017. Unsupervised query- focused multi-document summarization using the cross entropy method. In Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 961-964.
Acknowledgements
We would like to thank all the anonymous review- ers and the area chairs for their excellent review comments. This research is supported in part by the Natural Sciences and Engineering Research Coun- cil (NSERC) of Canada, the York Research Chairs (YRC) program, and the Generic Research Fund of York University. We also acknowledge Com- pute Canada for providing us with the computing resources. Jimmy Huang (jhuang @ yorku.ca) is the contact author of this paper.
Fabrizio Gilardi, Meysam Alizadeh, and Maël Kubli. 2023. Chatgpt outperforms corkesworkers for text- annotation tasks. arXiv preprint 230.::2333.50555
Travis Goodwin, Max Savery, and Dina Demner- Fushman. 2020. Flight of the pegasus? compar- ing transformers on few-shot and zero-shot multi- document abstractive summarization. In Proceedings of the 28th International Conference Computa- rional Linguistics, pages 5640-5646.
Biyang Guo, Xin Zhang, Ziyuan Wang, Mingi Jiang. Jinran Nie, Yuxuan Ding, Jianwei Yue, and Yupeng
hhttps://ogenai.commrrees

---

Wu. 2023. How close is chatgpt to human experts? comparison corpus, evaluation, and detection. arXiv preprint arXiv:2301.077 075977
Mike Lewis. Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy. Ves Stoyanov, and Luke 7ettlemoyer. 2019. BART: Denoising sequence-to-sequene
pre-training for nat- ural language generation, translation, and compre- hension. arXiv preprint arXiv: 191v:1910.13. 13461.
Pengcheng He. Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. Deberta: Decoding-enhance bert with disentangled attention. In International Confer- ence on Learning Representations.
Chin-Yew Lin. 2004. ROUGE: A package for auto- matic evaluation of summaries. In Text summariza- tion branches out, pages 74-81.
Tsrat Jahan, Md Tahmid Rahman Laskar, Chun Peng, and Jimmy Huang. 2023. A comprehensive eval- uation of large language models on benchmark biomedical text processing tasks. arXiv preprint arXiv:2310.04270
Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023. Gpteval: Nlg evaluation using gpt-4 with better human align- ment. arXiv preprint ar Xiv::33303 16634.
Jan Kocoñ, Igor Cichecki, Oliwier Kaszyca, Mateusz Kochanek, Dominika Szydlo, Joanna Baran, Julita Bielamiewicz, Marcin Gruza, Arkadiusz Janz, Kamil Kanclerz, al. 2023. Chatgpt: Jack of all trades, master none. arXiv preprint arXiv:2302. 10724.
Ramesh Nallapati, Bowen Zhou, Cicero dos Santos, Caglar Gulcehre, and Bing Xiang. 2016. Abstractive text summarization using sequence-to-sequenee rmns and beyond. Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning, pages 280-290.
Sayali Kulkarni, Sheide Chammas, Wan Zhu, Fei Sha, and Eugene le. 2020. Aquamuse: Automatically generating datasets for query-hased multi-document summarization. arXiv preprint wrXiv:2010.12694
Preksha Nema, Mitesh M Khapra, Anirban Laha, and Balaraman Ravindran. 2017. Diversity driven atten- tion model for query-based abstractive summariza- tion. In Proceedings of the 55th Anmual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers), pages 1063-1072.
Md Tahmid Rahman Laskar, M Saiful Bari, Mizanur Rahman, Md Amran Hossen Bhuiyan, Shafiq Joty. and Jimmy Huang. 2023a. systematic study and comprehensive evaluation of ChatGPT on benchmark datasets. In Findings of the Association for Com- putational Linguisties: ACL 2023, pages 431-469, Toronto, Canada. Association for Computational Lin- guistics.
R OpenAl. 2023. Gpt-4 technical report. arXiv, pages 2303-08774.
Md Tahmid Rahman Laskar, Enamul Hoque, and Jimmy Huang. 2020a. Query focused abstractive summa- rization via incorporating query relevance and trans- fer learning with transformer models. In Canadian Conference on Artificial Intelligence, pages 342-348. Springer.
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Car- roll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow in- structions with human feedback. arXiv preprint arXiv:2203.02155.
Md Tahmid Rahman Laskar, Enamul Hoque, and Jimmy Xiangji Huang. 2022. Domain adaptation with preitrdined transformers for query-focused ab- stractive text summarization. Computational Linguis- rics, 48(2):279-320.
Chengwei Qin, Aston Zhang, Zhuosheng Zhang, Jiaso Chen, Michihiro Yasunaga, and Diyi Yang. 2023. Is chatgpt general-purpose natural language process- ing task solver? arXiv preprint arXiv:2302.06: 06476.
Md Tahmid Rahman Laskar, Enamul Hoque, and Xi- angji Huang. 2020b. WSL-DS: Weakly supervised learning with distant supervision for query focused multi-document abstractive summarization. In Pro- ceedings of the 28th International Conference on Computational Linguistics, pages 5647-5654.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee. Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text trans- former. arXiv preprint arxiv: 106.0333...
Dan Su, Yan Xu, Tiezheng Yu, Farhad Bin Siddique, Elham J Barezi, and Pascale Fung. 2020. Caire- covid: a question answering and query-focused multi-document summarization system for covid-19 scholarly information management. arXiv preprint arXiv:2005.0397
Md Tahmid Rahman Laskar, Xiangji Huang, and Ena- mul Hoque. 2020c. Contextualized embeddings based transformer encoder for sentence similarity modeling answer selection task. In Proceedings of the 12th Language Resources and Evaluation Con- ference, pages 5505-5514.
Md Tahmid Rahman Laskar, Mizanur Rahman, Israt Jahan, Enamul Hoque, and Jimmy Huang. 2023b. Cqsumdp: A chnotpt-annotated resource for query- focused abstractive summarization based on debate- pedia. arXiv preprint arXiv:2305.066 06147.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Fukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in Neural Information Pro- cessing Systems, pages 5998-6008.

---

Jesse Vig, Alexander Fabbri, Wojciech Kryšcirsski Chien-Sheng Sheng Wu, and Wenhao Liu. 2021. Explor- ing neural models for query-focused summarization. arXiv preprint 122.0771
A
Appendix
A.1
Debatepedia Dataset Limitations
Based on randomly sampled 100 instances, it has been found in recent study (Laskar al., 2022, 2023b) that:
Jiaan Wang. Yunlong Liang, Fandong Meng, Haoxiang Shi, Zhixu Li, Jinan Xu, Jianfeng Qu, and Jie Zhou. 2023. Is chatgpt good nlg evaluator? a preliminary study, arXiv preprint arXiv:2303. 04048.
52% of the queries this dataset have no relevance to the documents or the summaries, as demonstrated in Table
Shuohang Wang, Yang Liu, Yichong Xu, Chenguang Zhu, and Michael Zeng. 2021. Want to reduce labeling cost? gpt-3 can help. arXiv preprint arXiv:2108 13487.
70% of the queries are close-ended (i.e., Yes/No type) questions (see Example 4 in Ta- ble 5).
Yizhong Wang, Kai Liu, Jing Liu, Wei He, Yajuan Lyu, Hua Wu, Sujian Li, and Haifeng Wang. 2018. Multi- passage machine reading comprehension with cross- passage answer verification. In Proceedings of the 56th Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), pages 1918 1927.
Though, many queries in this dataset are rel- evant to the documents but the summaries are more of generic due to shorter document length. The average length of the document in this dataset is only 66.4 words.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Rémi Louf, Morgan Fun- towicz, et al. 2019. HuggingFace's Transformers: State-of-the-ant natural language processing, ArXiv, abs/1910.03771.
In addition, many instances in this dataset only contain one word summary (see Example 2 in Table 5) for a given query that appears both in the train- ing and evaluation sets, which may also help the model to memorize such words for similar queries during the training phase. These issues may lead to an unexpected increase in the ROUGE score when the model starts learning to reproduce those words in the summary during the evaluation phase. Fur- thermore, we also find some instances where the length of the summary is longer than the document length, which usually happens in short documents (see Example 3 in Table 5).
Yumo Xu and Mirella Lapata. 2020. Coarse-to-fine -tefee
query focused multi-document summarization. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP),
pages 0632-3645
Yumo Xu and Mirella Lapata. 2021. Text sum- marization with latent queries. arXiv preprint
arXiv:2106.00104..
A.2 Example Prompt for Query Generation One example prompt to re-generate the query using LLMs is shown in Figure 1.
Xianjun Yang, Yan Li, Xinlu Zhang, Haifeng Chen, and Wei Cheng. 2023. Exploring the limits of chatgpt for query or aspect-based text summarization. arXiv preprint 230.02230..
A.3 Experimental Details
Jin Ln-Ge Yao, Xiaojun Wan, and Jianguo Xiao, 2017, Re- cent advances in document summarization. Knowl- edge and Information Systems, 53(2):297-336.
A.3.1 Models
To evaluate the effecttivnees of our ChatGPT anno- tated CQSumDP and PaLM annotated PQSumDP datasets, we fine-tune some state-of-the-ar pre- trained sequence to sequence models (Lewis et al., 2019; Raffel et al., 2019; Zhang et al., 2019; Good- win et al., 2020). For this purpose, we concatenate the query with the document and give as input to these models generate the query-focused abstrac- tive summaries this approach has shown impres- sive performance in the query-focused abstractive summarization task recently (Laskar et al., 2022). We describe these models below:
Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Pe- ter Liu. 2019. Pegasus: Pre-training with extracted gap-sentences for abstractive summarization. arXiv preprint 1xXiv:1112.08777
Ming Zhong, Da Yin. Tao Yu, Ahmad Zaidi, Mutethia Mutuma, Rahul Jha, Ahmed Hassan, al. 2021. Qmsum: new benchmark for query-basen based multi- domain meeting summarization. In Proceedings of the 2021 Conference of the North American Chap- ter of the Association for Computational Linguisties: Human Language Technologies, pages 5905-5921.

---

Example Query having no relevance with the document the summary.
Query: Does MBA enhance leadership skills?
Document: Business schools might improve your quantitative presentation and communication skills. It might but get you thinking about ethical and strategy. But two years of case studies aren' it go to turn you into a leader if you weren't died one. There's ' no learning charisma persuasiveness elegance or gut instinct.
Reference Summary: PhD will not improve em factors of leaders.
Example 2: One word summary having no relevance with the query or document.
Query: Education: do child benefit from watching tv?
Document: by watching news child learn about geography politics advances science everything simply and later explained. furthermore child learn about real-life situation that happens on everyday basis which will benefit them in future.
Reference Summary: News.
Example 3: The length of the summary is longer than the document with the query being irrelevant.
Query: activists where do the keys activists and organizations stand ?
Document: see an analyses of the article...
Reference Summary: philip martin berkeley davis and michael teitelbaum the mirage of mexican guest workers novidec # foreign affairs.
Example 4: More of close-ended question.
Query: friendships: does twitter harms relationships
Document: twitter helps those stay touches no matter how far they may be from cach other.
Reference Summary: long-distance friendships.
Table 5: Some examples demonstrating the limitations in the Debatepedia dataset.
BART (Bidirectional and Auto-Regressive Transformer): BART (Lewis et al., 2019) is pre-trained sequence-to-sequencc model based the encoder-decoder architecture that was pre- trained on large amount of diverse text data using the denoising anco-enc technique to recover the original form of corrupted document. The pre-training involved various objectives such as ro- tating the document, permuting sentences, infilling text, masking tokens, and deleting tokens. We use the pre-trained BART model since fine-tuning this model was found to be very effective in abstractive summarization (Laskar et al., 2022).
Pegasus (Pre-training with Extracted Gap- sentences for Abstractive Summarization): Pe- gasus (Zhang et al., 2019) is transformer-based pre-trained encoder-decoder model for abstractive summarization. Its pre-training objective involves generating summary like text from an input docu- ment. To achieve this, the PEGASUS model first selects and masks some sentences from the input (scument(s). It then concatenates these selected sentences to create a uses different approaches to select these sentences, such randomly selecting certain number of sentences. selecting the first few sentences, com- puting the ROUGE-I score between each sentence and the rest of the document to choose the top- scoring sentences. This pseudo-summary is then used for self-supervised learning. By pre-training on large datasets using this approach, the model achieves impressive fine-tuning performance on downstream summarization datasets.
The model
T5 (Text-to-Text Transfer Transformer): The T5 model (Raffel et al., 2019) is a transformer- based model based on the BERT architecture. How- ever, unlike traditional BERT models that classify input text into specific category, the T5 model treats all tasks such as text classification, question answering, neural machine translation, and text summarization sequencceto-sequuence problem using various pre-training objectives. After pre- training, the model fine-tuned on many down- stream tasks, achieving impressive performance across various datasets including summarization.
A.3.2 Implementation
We use the HuggingFace (Wolf et al., 2019)l to implement the baseline models for performance evaluation. Similar to the prior work, we concate-
"https:/huggiiggfacecco

---

Prompted Input LLM (ChatGPT/PalM)
document along its summary given Write down reasonable query relevant to document summary
traditions while the same time making progress society particularty economic progress. tracitiinaa values that favors discipline and autaromy for exam ple foster entreprencurial and econamic growth. families values acdition help fulfil individuals so that they can lead stable and unproductive lives. all these th ngs are key to sccial and economic stability end progress..
Document:
republi
see contradiction
Summary: republicans see contradictions between traditione and progress.
LLM generated query
What Republican perspective on balancing traditional values and progress society?
Figure Example Input to LLMs for Query Generation.
nated the query with the document to give input to the pre-trained baselines (i.e., BART, Pegasus, T5). The pre-trained model is then fine-tuned using NVIDIA V100 GPUs. The training batch size for BART was set to 16, while was set to for Pegasus and T5. The other hyperparameters were similar for all models, with the learning rate being set to 2e 3 3 and the maximum input (i.e., the con- catenated query and document) sequence length being 150 tokens. The minimum and the maxi- mum target (i.e., the generated summary) sequence lengths were 5 and 25, respectively. total of 10 epochs were run to fine-tune the pre-trained summa- rization models. We computed the ROUGE (Lin, 2004) scores in terms of ROUGE-1, ROUGE-2, and ROUGE-L using the Evaluate" library to com- pare the performance of different models on the respective test set. As noted earlier, for ChatGPT, we use the gpt-3. 5.t....botttbttttee -300 model; while for PaLM, we the text-bison@00 model.
ample, we find that even though the original query is descriptive, it does not have any relevance to the generated summary. Whereas both the ChatGPT and PaLM generated queries are very relevant both the document and the summary (in this ex- ample, PaLM generated query is more descriptive). For the third example, we find that the original query is related to "entrepreneurs". However, the document is about "product managers", not "en- trepreneurs". Meanwhile, the ChatGPT and PaLM generated queries also very relevant the doc- ument and both LLM-generated queries are the same. This analysis further demonstrates the rele- vance our LLM -LLM-eeeertted query in comparison to the original query in Debatepedia.
A.4 Qualitative Analysis of the Annotated Data
this section, we do some qualitative analyses between the queries in the Original Debatepedia dataset as well as the queries generated using LLMs in our proposed CQSumDP and PQSumDP ver- sions of the Debatepedia dataset. For our analysis, collect set of samples from this dataset and present them in Table 6. While comparing between the queries in the first example in the table, we find that the original query is just one word length and very ambiguous, while the ChatGPT generated query is more descriptive and more relevant to both the document and the summary. For the second ex-
"https: //wg://ngggggeeee ace. ctt co/spaces/ evaluate-metten mettic/roeer

---

Original Query
ChatGPT Query
PaLM Query
Source Document
Gold Summary
military:
What actions did the gov- ermment take improve the
What did obama do for veterans
provided better body provided the department of veterans af- fairs with than $ # # billion improve services america veter- ended media blackout sualties and the return fallen soldiers dover afb announced creation joint virtual lifetime electronic record for members of the u.s. armed forces improve quality medical care ended previous stop-loss policy that kept soldiers in irag/afghanistan longer than their enlistment date signed the veter- ans health care budget reform and trans- parency act authorizing advance appro- priations
troops
improved vices benefits and respect for troops.
situation for
U.S. troops and veterans?
department of veterans affairs by providing two-fisca year bud- get authority thus enabling hetter med- ical care for veterans endorsed by the american legion american veterans blinded veter ans association
economy has wto
Has NAFTA What
nafta and job losses" eyril morong PhD wall street journal may # # did nafta cause u.s. lose so many jobs citing figures provided million and # especially high- paying manufacture jobs probably not i say probahly since causality in any social science economics included difficult to prove since so many factors change so quickly the real world. but many high-paying manufacture jobs were lost it took many years until after nafta went into effect before they were but what about manufacture jobs ? we had just about # million in it actually rose to # million in # and was at # # in #.
nafta has de- creased the number
caused job
author say about effect of NAFTA on American wages?
benefited losses in the economy of
U.S?
range
american job
united states
entrepreneurs:
an MBA nec-
an nec- essary for prod- uct managers?
christopher cummings an nec- essary product managers ? product management meet pop culture. febru- ary # hindsight looking back the brass tacks of my mba experience were about the basics of management economics and business strategy could that have been picked up on the job ? mayhe
teach plan not just tactics
does an mha
essary for prod-
strategy
help en-
uct managers?
trepreneurs
however the more impor- tant throughline of the experience relates to critical thinking perspective and Iearn- ing when to lead and when to follow
I on the job especially as young pm can be easy to lose perspective to miss the forest for the trees at the time i was definitely into the plate-sping slate-spinning the go-go-go the tactics and day-to-day no time think: just keep moving. [... ] the mba experience
Table Comparisons between original queries and ALM--ennratee queries in some samples the Debatepedia dataset. Note that personally identifiable information this dataset is anonymized with token.


Reviews:
[{"review_id": "SHGOtFxoxe", "reviewer": "Reviewer_8uYK", "summary": "", "questions": "A) What are the statistics of the datasets after regenerating the queries?\n\nB) How much do the generated queries overlap with the reference summaries? It would be interesting to measure the similarity between them.\n\nC) Why, on the second method, did you regenerate the query for all the filtered samples instead of applying the relevance classifier and re-generate only those with a query-summary mismatch?\n\nD) Table 1. Do CQSumDP, PQSumDP, and Original Debatepedia have the same size?\n\nE) Tables 1 and 2 show marginal improvements over the original dataset that could not be statistically significant. Have you measured the confidence intervals of the results to validate the significance? They should be added to the tables to refute the improvements.\n\nF) In Section 3. What kind of queries do we find in Debatepedia dataset?\n\nG) Lines 373-379. How many queries did you check in total? Why is it not remarked in the main article content?\n\nH) Lines 310-311 and 387-391. Why did you write \"we ask three human evaluators\" when you evaluated those samples? It could derivate into misleading information. Also, it could carry a certain bias.\n\nI) Lines 309-318. What was your inter-agreement? It will be interesting to have that information.\n\nJ) Table 3, Example 1. Why do you state that the query and the summary are not relevant? The only mismatch between them seems to be MBA and PhD; the rest seems to be completely relevant.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank you for kindly reviewing the paper and providing your feedback. We have carefully read your reviews and prepared our responses accordingly to address your concerns. Our responses to your comments are stated below.\n\n**Response to Reasons to Reject:**\n\n**Regarding no study around LLM-based data sampling and no measurement of the performance of those models for that classification task:** In our submitted manuscript, we already conducted extensive experiments with our two approaches, the (i) unfiltered version: where only queries that are detected as not relevant were regenerated by LLMs while keeping the rest as intact, and the (ii) filtered version: where we sample some instances based on some rules and only regenerated the queries for the sampled instances while excluding the rest. We found that LLM-generated queries led to performance gains for both approaches in comparison to the original queries in the Debatepedia dataset. Our results are presented in Table 1 and Table 2. While our findings demonstrate that classifying by LLMs is not that useful for data sampling in comparison to rule-based sampling, the query regeneration by LLMs is found to be useful for performance improvement in both cases. To also measure the capability of LLMs in classifying whether the query is relevant to the document/summary, we sample 100 instances and evaluate using humans to know if they also agree with the classification done by LLMs. We find that the precision for the classification task for PaLM-2 is 71% while for ChatGPT is 66%. For the human evaluation, usually, at least 2 annotators did the evaluation, and if there was a disagreement, another annotator chimed and we considered the majority votes. \n\n**Regarding the nature and quality of the generated queries:** We have already shown some queries generated by different LLMs in Section A.3.6 in Table 7. Note that since we submitted to the short paper track, we had to demonstrate this in the appendix. Here, we also recap our analysis of the paper. \nFor our analysis, we collect a set of 3 samples from this dataset and present them in Table 7. While comparing between the queries in the first example in the table, we find that the original query is just one-word length and very ambiguous, while the ChatGPT-generated query is more descriptive and more relevant to both the document and the summary. For the second example, we find that even though the original query is descriptive, it does not have any relevance to the generated summary. Whereas both the ChatGPT and PaLM generated queries are very relevant to both the document and the summary (in this example, the PaLM generated query is more descriptive). For the third example, we find that the original query is related to “entrepreneurs”. However, the document is about “product managers”, not “entrepreneurs”. Meanwhile, the ChatGPT and PaLM generated queries are also very relevant to the document and both of these LLM-generated queries are the same. This analysis further demonstrates the relevance of our LLM-generated query in comparison to the original query in Debatepedia. For convenience, we also submitted our constructed dataset containing the regenerated queries during the paper submission which can be found in the supplementary material. \n\n**Regarding whether the methodology posed is fixing (maintaining all samples and fixing some of them) or filtering and re-generating a subset of the original dataset:** As mentioned in Section 3 of our paper, we proposed two approaches: (i) unfiltered version: maintaining all samples but only regenerate the queries by LLMs that are considered as problematic by LLMs, (ii) filtered version: exclude samples that seems irrelevant (e.g., very short documents which are even shorter than gold summaries, summaries that are more of close-ended QA: yes/no type, etc.) and only regenerate the queries by LLMs for the ones that are not excluded. \nIn our submitted manuscript, we already conducted extensive experiments with these two approaches and found that LLM-generated queries led to performance gain for both approaches in comparison to the original queries in the Debatepedia dataset. While we cannot directly compare these approaches based on their performance in the Debatepedia test set since the size of the dataset for the filtered version and unfiltered version is different, we find that in the MS-Marco dataset, the performance of models trained on the filtered version is superior than the unfiltered version (please see Table 2), establishing the effectiveness of our approach that utilizes rule-based technique for data sampling followed by regenerating the queries for the sampled instances using LLMs. \n\n**Regarding the size of the generated datasets (CQSumDP, PQSumDP):** We have already mentioned this in line 214-216 of our submitted manuscript: the filtered versions size: 5212/301/401 instances in training, validation, and test sets while for the unfiltered datasets: 12000/719/1000 in training, validation, and test sets, respectively.\n\n**Regarding superficial data results analysis:** While we submitted a short paper, we already demonstrated our analysis on the in-domain dataset: Debatepedia and domain generalization on the MS-MARCO dataset. We also presented human evaluation of the LLM-Generated queries in the main paper. Meanwhile, due to the restriction of 4 pages in the short paper track, we presented the results based on model scaling, ablation studies, zero-shot learning, and qualitative analysis of the annotated dataset in the Appendix (Section A.3.3, A.3.4, A.3.5, A.3.6, lines 725 to 817). It should be noted that since the accepted papers will get one additional page, we can also demonstrate some of these findings presented in the appendix in the main paper. \n\n**Regarding whether re-generating the queries with an LLM that simultaneously sees the document and the summary could generate queries excessively biased towards the reference summaries:** Thanks for pointing this out. We have computed this for all types of queries: original queries, PaLM-generated queries, as well as ChatGPT-generated queries, and find that the frequency of words that appear both in queries and summaries are 17% when original queries were used, 17% when generated by PaLM-2, and 19% when generated by ChatGPT. Thus, LLM-generated queries did not have any significantly high overlaps with the words in the reference summaries. More importantly, we find that the LLM-based query regeneration approach, especially the one based on ChatGPT, is generally better for not only the Debatepedia dataset but also for the out-of-domain dataset: the MS-MARCO dataset. This further mitigates any concerns of bias towards the reference summaries.\n\n**Questions For The Authors:**\n\n**What are the statistics of the datasets after regenerating the queries?**\nWe have mentioned this in lines 214-216 of our submitted manuscript. The filtered version size: 5212/301/401 instances in training, validation, and test sets, while the unfiltered version size: 12000/719/1000 in training, validation, and test sets, respectively.\n\n**How much do the generated queries overlap with the reference summaries? It would be interesting to measure the similarity between them.**\nWe have computed this for both original queries, PaLM-generated queries, as well as ChatGPT-generated queries and find that the frequency of words that appear both in queries and summaries are 17% when original queries were used, 17% when generated by PaLM-2, and 19% when generated by ChatGPT. \n\n**Why, on the second method, did you regenerate the query for all the filtered samples instead of applying the relevance classifier and re-generate only those with a query-summary mismatch?** This is done to compare our two proposed approaches: (i) whether we can solely rely on LLMs for both relevancy classification and query regeneration, or (ii) can we sample instances via rule-based techniques designed by humans to address existing issues in Debatepedia and only re-generate the queries using LLMs for the sampled instances. So in our first approach, we have already done this: “applying the relevance classifier and re-generate only those with a query-summary mismatch?”. On the second approach, we address existing issues in Debatepedia (e.g., short documents that are even shorter than gold reference summaries, summary length of only 1-2 words that are more of close-ended question-answering, etc.) by removing instances that are less relevant for query-focused summarization and only re-generate the queries for the filtered instances.  \n\n**Table 1. Do CQSumDP, PQSumDP, and Original Debatepedia have the same size?** We have mentioned this in lines 214-216 of our submitted manuscript. The filtered version size: 5212/301/401 instances in training, validation, and test sets, while the unfiltered version size: 12000/719/1000 in training, validation, and test sets, respectively. So the respective versions (filtered and unfiltered) will have the same size for different annotations (LLM and Original): CQSumDP, PQSumDP, Original Debatepedia. Thus, the filtered versions size for CQSumDP, PQSumDP, and Original Debatepedia are 5212/301/401 instances in training, validation, and test sets; while for the unfiltered datasets, the sizes for CQSumDP, PQSumDP, Original Debatepedia are: 12000/719/1000 in training, validation, and test sets, respectively.\n\n**Tables 1 and 2 show marginal improvements over the original dataset that could not be statistically significant. Have you measured the confidence intervals of the results to validate the significance? They should be added to the tables to refute the improvements.** As we submitted to the short paper track based on our early findings, unfortunately, we did not conduct any significance tests in our initial manuscript. Meanwhile, the test set of filtered and unfiltered versions for Debatepedia are different, and so we can not directly compare these two versions on Debatepedia to measure whether the performance gains are significant or not. However, since we utilize MS-MARCO for domain generalization performance evaluation, the evaluation datasets are the same in this case. Thus, we conduct significance tests for the filtered and unfiltered versions of CQSumDP, PQSumDP, and Original Debatepedia when these datasets are used for model training followed by evaluation on MS-MARCO.\nBased on paired t-test, we find that the filtered versions for each model significantly (**p ≤ .05**) outperform the unfiltered versions in terms of all ROUGE metrics. However, the difference is not statistically significant in terms of BERTScore. Note that for MS-MARCO, the ROUGE metric, especially ROUGE-L is the more standard metric, as used in the following papers:\n\n*Laskar, M. T. R., Hoque, E., & Huang, J. X. (2022). Domain adaptation with pre-trained transformers for query-focused abstractive text summarization. Computational Linguistics, 48(2), 279-320.*\n\n*Nishida, K., Saito, I., Nishida, K., Shinoda, K., Otsuka, A., Asano, H., & Tomita, J. (2019, July). Multi-style Generative Reading Comprehension. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (pp. 2273-2284).*\n\n**In Section 3. What kind of queries do we find in the Debatepedia dataset?** As mentioned in lines 91-95, the Debatepedia dataset requires generating summaries from a document containing argumentative text (i.e., arguments and counter-arguments) for the given query. We have also presented the limitation of existing queries in Debatepedia in Section A.1 (lines 611 to 638):\n- 52% of the queries in this dataset have no relevance to the documents or the summaries, as demonstrated in Table 3.\n- 70% of the queries are close-ended (i.e., Yes/No type) questions (see Example 4 in Table 3).\n\n**Lines 373-379. How many queries did you check in total? Why is it not remarked in the main article content?** Thanks for pointing this out. Note that we have checked all the queries in the filtered versions, consisting of 5212/301/401 queries in the training/validation/test sets, respectively. We will mention this in the main article in the next iteration of the paper. \n\n**Lines 310-311 and 387-391. Why did you write \"we ask three human evaluators\" when you evaluated those samples? It could derivate into misleading information. Also, it could carry a certain bias.** We will fix this accordingly. Note that the evaluation of these LLMs were done by the authors since we were only evaluating the queries generated by two of the existing LLMs that we study in this paper. We did not propose any new LLMs in this paper and so there are no risks of any potential biases while doing this evaluation. To further ensure high quality evaluation and mitigate any risks of potential biases, the human evaluators did not have any information on which queries were generated by which LLMs. Meanwhile, evaluations of LLMs via paper authors have also been done in recent months. For instance, human evaluation of LLMs by authors have been done in the following paper in ACL 2023: \n\n*Md Tahmid Rahman Laskar, M Saiful Bari, Mizanur Rahman, Md Amran Hossen Bhuiyan, Shafiq Joty, and Jimmy Huang. 2023. A Systematic Study and Comprehensive Evaluation of ChatGPT on Benchmark Datasets. In Findings of the Association for Computational Linguistics: ACL 2023, pages 431–469, Toronto, Canada. Association for Computational Linguistics.*\n\n**Lines 309-318. What was your inter-agreement? It will be interesting to have that information.** Using Cohen’s Kappa, we obtain good agreement, 0.6 < k <= 0.8 between annotators.\n\n**Table 3, Example 1. Why do you state that the query and the summary are not relevant? The only mismatch between them seems to be MBA and PhD; the rest seems to be completely relevant.** We state this because the query wanted to know whether the MBA degree can enhance specific skills but the summary rather generated something related to another degree. Thus, we consider this query-summary pair as irrelevant. While we agree that the rest seems relevant, still there are discrepancies between what was required in the query and what was in the summary. \n\n**Typos Grammar Style And Presentation Improvements:** We will fix all the typos that are stated. \n\n**Reproducibility Concern:** Regarding the reproducibility concern, we would like to inform that to ensure reproducibility, we have already included the dataset in the supplementary material. Moreover, we have already mentioned the computing resources we used for experiments, alongside hyperparameters and the model details, etc. in Section A.3.3 in the paper (please see line: 701-724). Meanwhile, the dataset will also be made publicly available upon acceptance of the paper, as mentioned in the submitted manuscript.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Exploring the usage of Large Language Models for improving/fixing data in existing datasets or creating new datasets is a clear research path. This work defines a good case of study around the Debatepedia dataset, where some samples suffer from a specific problem, such as query-summary mismatching.", "reasons_to_reject": "This work presents some weaknesses regarding the information provided:\n     \n- There is no study around LLM-based data sampling, where Chat-GPT or PaLM-2 detects which queries should be rewritten. There is no measurement of the performance of those models for that classification task.\n\n- There is no data on the nature and quality of the generated queries.\n\n- It is unclear whether the methodology posed is fixing (maintaining all samples and fixing some of them) or filtering and re-generating a subset of the original dataset.\n\n- The size of the generated datasets (CQSumDP, PQSumDP) is unclear. Thus, the results presented could also be unclear.\n\n- Superficial data results analysis.\n\nAlso, a big concern about the methodology: re-generating the queries with an LLM that simultaneously sees the document and the summary could generate queries excessively biased towards the reference summaries that could help the summarization models.", "questions_to_authors": "A) What are the statistics of the datasets after regenerating the queries?\n\nB) How much do the generated queries overlap with the reference summaries? It would be interesting to measure the similarity between them.\n\nC) Why, on the second method, did you regenerate the query for all the filtered samples instead of applying the relevance classifier and re-generate only those with a query-summary mismatch?\n\nD) Table 1. Do CQSumDP, PQSumDP, and Original Debatepedia have the same size?\n\nE) Tables 1 and 2 show marginal improvements over the original dataset that could not be statistically significant. Have you measured the confidence intervals of the results to validate the significance? They should be added to the tables to refute the improvements.\n\nF) In Section 3. What kind of queries do we find in Debatepedia dataset?\n\nG) Lines 373-379. How many queries did you check in total? Why is it not remarked in the main article content?\n\nH) Lines 310-311 and 387-391. Why did you write \"we ask three human evaluators\" when you evaluated those samples? It could derivate into misleading information. Also, it could carry a certain bias.\n\nI) Lines 309-318. What was your inter-agreement? It will be interesting to have that information.\n\nJ) Table 3, Example 1. Why do you state that the query and the summary are not relevant? The only mismatch between them seems to be MBA and PhD; the rest seems to be completely relevant.", "paper_topic_and_main_contributions": "In this paper, the authors present a methodology for fixing queries of query-focused abstractive summarization that do not match the reference summaries of the document-query-summary samples. They exemplify this methodology with the Debatepedia dataset. ChatGPT and Palm-2 are used for regenerating the samples' queries and creating new versions of the Debatepedia dataset. Later, (BART, PEGASUS, and T5)-based query-focused abstractive summarization models are used to measure the impact on the performance after rewriting the queries with LLMs. The results show an improvement in the performance of some summarization models.", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "Typos/Style:\n\n- 66-67: whether SOTA LLMs can be utilized\n\n- 68: Our experiments\n\n- 145: we\n\n- 147: and regenerated these queries\n\nSuggestions:\n\n- 225-227: You can not ask to generate a query and summary giving a document since this would be a completely (and synthetic) dataset. It is not because of the length of the summaries.\n\n- 326-330: Having some reference that refutes this affirmation would be nice."}}, {"review_id": "DWsy4lVYGJ", "reviewer": "Reviewer_2X9Y", "summary": "", "questions": "What are the inter-annotator agreement and the specific setting of the human evaluation you conducted for comparing the quality of queries written by ChatGPT and PaLM?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank you for thoroughly reading the paper and providing valuable feedback. We have carefully read your reviews and prepared our responses accordingly to address your concerns. Our responses to your questions are stated below.\n\n**Response to Reasons to Reject:**\n\n**Reason 1:** Though we already did the domain generalization evaluation on the MS-MARCO dataset to demonstrate the effectiveness of LLM-generated queries, we also put the results for the in-domain Debatepedia dataset as we found similar analysis on the in-domain dataset in the literature for datasets related issues. For instance:\n\n*Guo, Y., Clavel, C., Eddine, M. K., & Vazirgiannis, M. (2022, December). Questioning the Validity of Summarization Datasets and Improving Their Factual Consistency. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (pp. 5716-5727).*\n\n*Laskar, M. T. R., Chen, C., Fu, X. Y., & Tn, S. B. (2022, December). Improving Named Entity Recognition in Telephone Conversations via Effective Active Learning with Human in the Loop. In Proceedings of the Fourth Workshop on Data Science with Human-in-the-Loop (Language Advances) (pp. 88-93).*\n\n**Reason 2:** We initially reported the results based on only the ROUGE metric as this is mostly done when the results are reported for Debatepedia [1, 2]. Moreover, since we conducted extensive experiments in different scenarios, filtered and unfiltered versions, on both Debatepedia for in-domain evaluation as well as MS-MARCO for domain generalization, conducting human evaluation on the outputs of all models will require significant time. However, based on your suggestions to conduct a qualitative evaluation, we utilize GPT-4 to conduct human-like evaluation due to its impressive performance as an evaluator across various tasks, including state-of-the-art performance on the summarization task as a human-like evaluator [3]. \n\nIn our human-like qualitative evaluation using GPT-4, we ask GPT-4 to score between 1 to 5 for the factual consistency and the coherence of the summaries generated by different models, as generating a factually correct summary as well as maintaining coherence during summary generation two of the important metrics that may depend on the given queries. \n\nWe sample 10 summaries generated by each model on the MS-MARCO dataset and compute the average score for the filtered version and the unfiltered version (our results are stated below). We find that the factual consistency and the coherence score of the models are much better in the filtered version in comparison to the unfiltered version:\n\nCQSUMDP (filtered): \n- Coherence: 2.87\n- Factual Consistency: 3.00\n\nCQSUMDP (unfiltered): \n- Coherence: 1.60\n- Factual Consistency: 1.01\n\nPQSUMDP (filtered): \n- Coherence: 3.30\n- Factual Consistency: 2.93\n\nPQSUMDP (unfiltered): \n- Coherence: 1.27\n- Factual Consistency: 1.00\n\nOriginal Debatepedia (filtered): \n- Coherence: 2.87\n- Factual Consistency: 2.87\n\nOriginal Debatepedia (unfiltered): \n- Coherence: 1.33\n- Factual Consistency: 0.98\n\nReferences:\n\n[1] Preksha Nema, Mitesh M. Khapra, Anirban Laha, and Balaraman Ravindran. 2017. Diversity driven attention model for query-based abstractive summarization. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1063–1072, Vancouver, Canada. Association for Computational Linguistics.\n\n[2] Md Tahmid Rahman Laskar, Enamul Hoque, and Jimmy Xiangji Huang. 2022. Domain Adaptation with Pre-trained Transformers for Query-Focused Abstractive Text Summarization. Computational Linguistics, 48(2):279–320.\n\n[3] Liu, Yang, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. \"Gpteval: Nlg evaluation using gpt-4 with better human alignment.\" arXiv preprint arXiv:2303.16634 (2023).\n\n**Reason 3:** We did not do that initially because such human evaluations in the original queries are already conducted in earlier research, which we cited in the paper [1] as well as specifically mentioned in lines 611-620. Based on your suggestion, we conducted human evaluation again on a randomly sampled 100 instances. But this time, we ask the annotators to select the preferred queries from Original Queries, ChatGPT-generated, and PaLM-generated. We find that the ChatGPT-generated queries were preferred 56% times, PaLM-generated ones 32% times, and the original queries 12% times. \n\n[1] Md Tahmid Rahman Laskar, Enamul Hoque, and Jimmy Xiangji Huang. 2022. Domain Adaptation with Pre-trained Transformers for Query-Focused Abstractive Text Summarization. Computational Linguistics, 48(2):279–320.\n\n**Questions for the Authors:**\n\n**What are the inter-annotator agreements and the specific setting of the human evaluation you conducted for comparing the quality of queries written by ChatGPT and PaLM?** Using Cohen’s Kappa, we obtain good agreement, 0.6 < k <= 0.8 between annotators. The annotators were required to select the preferred query based on considering the following: (i) relevance with the summary, (ii) relevance with the document, and (iii) conciseness.\n\n**Reproducibility Concern:** Regarding the reproducibility concern, we would like to inform that to ensure reproducibility, we have already included the dataset alongside the supplementary material. Moreover, we have already mentioned the computing resources we used for experiments, alongside hyperparameters and the model details, etc. in Section A.3.3 in the paper (please see lines: 701-724). Meanwhile, the dataset will be made publicly available upon acceptance of the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper provides a case study of leveraging LLMs for cleanling noisy datasets. This idea in general can have great potential.", "reasons_to_reject": "The experimental/evaluation setting of this paper cannot sufficiently support the claim of this work. Specifically, the authors give two reasons for arguing the effectiveness of LLM-based data cleaning:\n- The models that are trained on the cleaned dataset can achieve better performance in automatic evaluation on the *same* dataset that is used for model training. I'm not sure why this can support the claim that the LLM-cleaned dataset is better, since the performance of the model trained on a dataset is not a direct measurement of the quality of the same dataset. \n- The models trained on the LLM-cleaned dataset show better performance on other datasets which demonstrates better generalization ability.  This is a more convincing point than the first one, however, the improvement seems to be marginal under automatic evaluation and the authors did not perform human evaluation to compare the performance of the models trained on the original dataset and the cleaned dataset. \n- While the authors provide a human evaluation for comparing the quality of generated queries from ChatGPT and PaLM, they did not provide a similar study for comparing the query quality of the original dataset and the LLM-cleaned dataset, which in my opinion would be the clearest way to demonstrate the effectiveness of the LLM-based data clean method proposed in this work.", "questions_to_authors": "What are the inter-annotator agreement and the specific setting of the human evaluation you conducted for comparing the quality of queries written by ChatGPT and PaLM?", "paper_topic_and_main_contributions": "This paper investigates the task setting of leveraging large language models (LLMs) to clean noisy datasets. The authors use two approaches mostly for this purpose: (1) a rule-based method for filtering out low-quality examples; (2) using LLMs to rewrite the queries for query-focused text summarization. Using Debatepedia as the example dataset, the authors show that the dataset cleaned by LLMs is able to help the models to achieve better performance on the dataset that is used for fine-tuning and other related datasets under automatic evaluation.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "phoUyR8oFG", "reviewer": "Reviewer_UJ5T", "summary": "", "questions": "1. Could you provide some additional insights into the qualitative differences of summaries trained with and without LLM-annotated data?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank you for thoroughly reading the paper and providing constructive feedback. We have carefully read your reviews and prepared our responses accordingly to address your concerns. Our response to your comments is stated below.\n\n**Response to Reasons to Reject:**\n\n**The paper is rather narrow in scope, only applying the method to a single dataset, while the approach could have larger implications to other query-focused datasets or noisy datasets in general:** While we agree that the paper is quite narrow in score, this is also the primary reason for submitting to the short paper track as the early-stage findings of this paper are more relevant to the requirements of short papers. As the Debatepedia dataset has existing limitations in terms of query relevance, we believe that the solution proposed by us will not only be useful for researchers working in this area but can also be extended to other datasets and tasks.\n\n**Questions for the Authors:**\n\nWe initially reported the results based on only the ROUGE metric as this is mostly done when the results are reported for Debatepedia [1, 2]. Since we conducted extensive experiments across various models (BART, T5, PEGASUS) in different scenarios, filtered and unfiltered versions, on both Debatepedia for in-domain evaluation as well as MS-MARCO for domain generalization, conducting human evaluation on the outputs of all models will require significant time. Thus, we utilize GPT-4 to conduct human-like qualitative evaluation due to its impressive performance as evaluators across various tasks, including state-of-the-art performance on the summarization task as human-like evaluators [3]. \n\nIn our human-like qualitative evaluation using GPT-4, we ask GPT-4 to score between 1 to 5 for the factual consistency and the coherence of the summaries generated by different models, as generating a factually correct summary as well as maintaining coherence during summary generation are two of the important metrics that may depend on the given queries. \n\nWe sample 10 summaries generated by each model on the MS-MARCO dataset and compute the average score for the filtered version that is based on LLM-generated queries and the unfiltered version that is based on the original queries. Our results are stated below. We find that the factual consistency and the coherence score of the models are much better in the filtered version in comparison to the unfiltered original version:\n\nCQSUMDP (filtered): \n- Coherence: 2.87\n- Factual Consistency: 3.00\n\nPQSUMDP (filtered): \n- Coherence: 3.30\n- Factual Consistency: 2.93\n\nOriginal Debatepedia (unfiltered): \n- Coherence: 1.33\n- Factual Consistency: 0.98\n\nReferences: \n\n[1] Preksha Nema, Mitesh M. Khapra, Anirban Laha, and Balaraman Ravindran. 2017. Diversity driven attention model for query-based abstractive summarization. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1063–1072, Vancouver, Canada. Association for Computational Linguistics.\n\n[2] Md Tahmid Rahman Laskar, Enamul Hoque, and Jimmy Xiangji Huang. 2022. Domain Adaptation with Pre-trained Transformers for Query-Focused Abstractive Text Summarization. Computational Linguistics, 48(2):279–320.\n\n[3] Liu, Yang, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. \"Gpteval: Nlg evaluation using gpt-4 with better human alignment.\" arXiv preprint arXiv:2303.16634 (2023).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper addresses a well-known issue with an existing dataset and shows how to efficiently leverage LLMs to correct it. The approach is simple and does improve performance. ", "reasons_to_reject": "1. The paper is rather narrow in scope, only applying the method to a single dataset, while the approach could have larger implications to other query-focused datasets or noisy datasets in general.\n2. The paper only briefly mentions a human evaluation (a bit more discussed in the appendix) of the queries produced. A comparison of the summaries produced with and without the LLM-annotated data would be insightful.", "questions_to_authors": "1. Could you provide some additional insights into the qualitative differences of summaries trained with and without LLM-annotated data?", "paper_topic_and_main_contributions": "The paper is about using LLMs (ChatGPT and PaLM) to improve the Debatepedia dataset to make it more suited for query-focused abstractive summarization. Recent work has found that many of the queries in the dataset have no relevance with the source document, and the paper proposes two approaches for data sampling to identify and regenerate these queries. Models fine-tuned on the LLM-annotated dataset usually perform better than those trained on. ", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Tables 1 and 2 are hard to read without bolding the top performing systems."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper investigates how we can use LLMs to clean noisy Debatepedia for query-focused summarization using rule-based filtering and query rewriting. Reviewers agree that the experiments have demonstrated the effectiveness of the proposed approach, and this could have potential impacts if we can use LLM to fix annotation errors for broader tasks. The author response has clarified the reason why it reports both in-domain and out-of-domain evaluation and additional GPT-4 based and human evaluation to show the improvements and query quality. The paper is narrow in scope which limits its excitement. In addition to the new results, we also strongly encourage authors to incorporate the feedback to improve the paper clarity.
Decision: Accept-Findings