Select, Prompt, Filter: Distilling Large Language Models for Summarizing Conversations
Authors: Minh-Quang PHAM
Sathish Reddy Indurthi
Shamil Chollampatt
Marco Turchi
Venue: EMNLP 2023 Main (2023)
Keywords: Text Summarization
Generative AI
Knowledge distillation
Data filtering
TL;DR: Data synthesis for summarizing conversations

Abstract: Large language models (LLMs) like ChatGPT can be expensive to train, deploy, and use for specific natural language generation tasks such as text summarization and for certain domains. A promising alternative is to fine-tune relatively smaller language models (LMs) on a particular task using high-quality, in-domain datasets. However, it can be prohibitively expensive to get such high-quality training data. This issue has been mitigated by generating weakly supervised data via knowledge distillation (KD) of LLMs. We propose a three-step approach to distill ChatGPT and fine-tune smaller LMs for summarizing forum conversations. More specifically, we design a method to selectively sample a large unannotated corpus of forum conversation using a semantic similarity metric. Then, we use the same metric to retrieve suitable prompts for ChatGPT from a small annotated validation set in the same domain. The generated dataset is then filtered to remove low-quality instances. Our proposed select-prompt-filter KD approach leads to significant improvements of up to 6.6 ROUGE-2 score by leveraging sufficient in-domain pseudo-labeled data over a standard KD approach given the same size of training data.

Select, Prompt, Filter: Distilling Large Language Models for Summarizing Conversations
Minh-Quang Pham; Sathish Reddy Indurthi;
Shamil Chollampatt, Marco Turchi
Zoom Video Communications
mi www.pppwwaoo....oo mi sathi
us,
shamil www.echhhhllllpamcccccocccc chhlllammattteoooo us, marco. wwwrchiizoo....mm us
Abstract
VALID
RAW DATABET
Model
Large language models (LLMs) like ChatGPT can be expensive train, deploy, and use for specific natural language generation tasks such as text summarization and for certain domains. A promising alternative fine-tune relatively smaller language models (LMs) particu- lar task using bigh-quality, in-domain datasets. However, it can be prohibitively expensive to get such high-quality training data. This issue has been mitigated by generating weakly su- pervised data via knowledge distillation (KD) of LLMs. We propose a three-step approach distill ChatGPT and fine-tune smaller LMs for summarizing forum conversations. More specifically, we design a method to selectively sample large unannotated corpus of forum conversation using a semantic similarity met- ric. Then, we use the same metric to retrieve suitable prompts for ChatGPT from small notated validation set in the same domain. The generated dataset then filtered to remove low- quality instances. Our proposed pelept-prompt- filter KD approach leads to significant improve- ments up 6.6 ROUGE-2 score by leverag- ing sufficient in-domain pseudo-lahelled data, over a standard KD approach given the same size of training data.
Staduning
SELECT
DATASET
PROMPT
ChatGPT
Figure 1: Schematic diagram of select-prompt-fiter
knowledge distillation.
et al., 2022; Li et al., 2023). An alternate strategy is to fine-tune relatively much smaller language mod- els (LM) like BART (Lewis et al., 2020) or T5 (Raf- fel et al., 2020) on the target task. These fine-tuned models, however, require large annotated training datasets to achieve competitive results, preventing the adoption of this approach for wide variety of tasks and domains. Obtaining sufficient annotated data from human annotators can also be substan- tially expensive (Wang et al., 2021).
An approach to address this issue of limited high-quality annotated data involves creating syn- thetic data that extracts and distills knowledge from ICL-capable LLMs smaller specialized models. This technique of knowledge distillation (Hinton et al., 2015; Kim and Rush, 2016) helps specialize smaller pre-trained LMs to particular tasks using large amounts of such synthetic annotations. For example, Wang et al. (2021) use GPT-3 to annotate training data for nine NLP tasks, including summa- rization, estimating up to 96% reduction in costs using crowdsourcing and achieving similar performance. Ding al. (2022) GPT-3 get labels and even generate data from scratch for quence tagging tasks. However, previous work on distillation addresses the data annotation process without careful data selection, prompt engineering,
1 Introduction
Large language models (LLMs)) such as the GPT- series models have demonstrated great strengths in a range of natural language understanding and gen- eration tasks with their ability to do few-shot or in- context learning (ICL, Brown et al., 2020). In this scenario, the input for desired task is enhanced by incorporating few examples (or demonstrations) of that particular task given the prompt to the LLM. This helps to customize and optimize the LLM inference time, leading to higher-quality results. Many of these LLMs, however, are astro- nomically expensive and environmentally unsus- tainable to train and use (Bender et al., 2021; Wu
"qqua contribution

---

data filtering, leading to suboptimal results. To get a better-distilled model and optimize the cost of synthetic data annotation, we propose a sys- tematic distillation approach, named select-prompt- filter knowledge distillation (SPF KD), for summa- rization. We use ChatGPT as the ICL-capable LLM that we distill. In this paper, we focus on summariz- ing online messaging conversations (e.g., forums) for which auman-annotated datasets are not pub- licly available, though our method can be applied to other low-resource domains as well. First, a suit- able subset of conversations is selected for distilla- tion from large unannotated raw corpus, based on similarity to small annotated validation set. Then, for each conversation in the selected subset, seman- tically similar conversations are retrieved from the validation set and sent to ChatGPT as the prompt to get its synthetic summary. Liu et al. (2022a) use a similar prompt-retrieval approach based on various sentence embeddings to improve GPT-3 performance on downstream tasks, although not in the context of synthetic data annotation. Thereafter, we apply filter on the generated conversation- summary pairs remove spurious and low-quality annotations. Figure shows a schematic overview of our approach. Our proposed method uses a semantic similarity score (Cer et al., 2018) and a reference-less evaluation measure (Egan et al., 2022) at different steps. We evaluate our approach on three forum and email conversation test sets. We show that the proposed KD method is able to sig- nificantly improve summarization performance of up 6.6 ROUGE-2 scores compared to a standard KD approach.
Fundamental to our approach are two metrics that we use in the above steps:
USE-Cosine:
We use Universal Sentence Encoder' (USE, Cer et al. 2018) to get the embeddings of a conversation given as text string. USE inter- nally uses a Deep Averaging Network (lyyer et al., 2015) that averages the embeddings of the words and bigrams and passes it through a feed-forward network. We compute the cosine distance between the two conversation embeddings. We refer to this metric as USE-cosine, which we use for computing connersation-comeessiee
similarity in the selee- tion and prompt retrieval steps.
Shannon Score: Shannon Score? (Egan et al.. 2022) is a reference-free summary evaluation met- ric that measures the difference in the ability of a language model to generate the conversation C. with and without the summary S. Specifically, GPT-2 is used compute the information difference, i.c., the difference between log probabilities of generating given S as the prompt and generating C. without it. The final metric, Shannon Score (SS), is the ratio of this value to the information difference when the C itself is used as the summary. This is under the assumption that the conversation conveys the entire information and thus the denominator becomes the upper bound of the information difference.
log P(C]S) log P(C)
SS(C,S) =
(1)
log P(CIC) - log P(C)
We use Shannon Score to compute the relevance and filter the synthetic annotations generated by ChatGPT that do not satisfy a certain threshold.
conversation--sumnaal
In our proposed approach, we rely on the avail- ability of a large unannotated training dataset of forum threads (D₁) for knowledge distillation for our target task. We use a large corpus of forum conversations for this. We also assume the avail- ability of small annotated validation set D, with M conversation summary pairs. We describe the three steps used in our approach in greater detail below.
2 Approach
Our method addresses the lack of high-quality annotated summarization datasets certain do- mains fine-tune smaller summarization models via sequence-level knowledge distillation (Kim and Rush, 2016). We propose three-step approach to get the annotated training data from ICL-capable LLMs like ChatGPT, which acts as our teacher model that we distill. The three steps (Figure 2) in our proposed select-prompt-file filter (SPF) knowl- edge distillation approach are (i) data selection (ii) prompt retrieval, and (iii) data filtering. These steps are done to get high-quality synthetic sum- maries for the target task of summarizing conver- sations. The synthetic dataset will then be used to fine-tune a smaller pretrained LM (student model).
2.1 Data Selection
To control costs when using ChatGPT for annota- tion, we use a similarity-bant based selection approach to select N conversations from D, for generating synthetic summaries. As shown in Figure 2 (left),
htttp:://ffuub.de//gggg universal- mniverrsl--eetteneeencccdrrr
Phttps://gii thub. com/PrrimerA//llanc//

---

Data Selection
Prompt Retrieval
Data Filtering
Traning
Selected Training
Conversation Summary
top-A
Top-3
Valideeoon Set
Mildesaa Set
ChatGPT
Figure 2: The three steps of our proposed approach: data selection, (ii) prompt retrieval, and (iii) data filtering.
for each validation conversation in D. we select top & conversations (A: = N/M) from Dr without replacement using the USE-cosine similarity. In this way, ensure that represent every exam- ple of the validation set independently to capture maximum variance in the training data, N deter- mined based on the available budget. This selected training data is denoted by D,, which is a subset of Dr-
Experimental Setup
3.1 Data
We use conversations from StackExchange fo- rums², except non-Einglish language forums and StackOverflow due to size, as the raw unanno- tated dataset. For validation and testing, we use the Ubuntu and NYC forum summarization cor- pora (Bhatia et al. 2014, 100 instances each), and the BC3 email summarization corpus (Ulrich et al. 2008, 40 instances). We divide each dataset equally for validation and testing. We combine the three validation sets into a single validation set of 120 instances.
2.2 Prompt Retrieval
For each conversation in the selected training data Da, we apply similarity- -based prompt retrieval technique similar to Liu et al. (2022c) to construct the prompt. We select one or more examples from the Dr that are closest to each conversation in D, according to the USE-cosine metric (see Figure 2, center). The retrieved examples along with the cor- responding training conversation to be summarized are passed to ChatGPT as the prompt to get the summary (sec example prompt in Appendix A. Thus, we get an annotated training set D which is a set of forum threads and their corresponding ChatGPT generated summaries.
We use OpenAl ChatGPT our teacher model for distillation. We restrict to selecting 100K con- versations for ChatGPT annotation from StackEx- change to demonstrate a limited budget scenario. Annotating 100K conversations costs about 200 USD+ 3. 5-turbo) assuming each annota- tion input is under 1000 tokens. Our baseline is standard knowledge distillation (Standard KD) approach, where a random 100K subset is selected from StackExchange for distillation. For the base- line, we select two random examples as the prompt from our validation set to be sent to ChatGPT for getting synthetic summaries and we do not filter the results. During the data selection step of our proposed SPF KD approach, for each conversation in our validation set, we select k 833 exam- ples (000000 833) from StackExchange based on 120
2.3 Data Filtering
Some of the ChatGPT-gennrated summaries in D can be of poor quality and diverge from the con- versation significantly. We filter out such spurious examples that do not satisfy the certain threshold of Shannon Score, get a smaller annotated train- ing dataset Di, (see Figure 2, right). Dj is the distilled training data from the teacher that is then used to finetune a student model specialized for summarizing conversations.
USE-cosine similarity (Section 2.1). For each lected conversation, we prompt ChatGPT with two examples from the validation set that are closest to it based on the USE-Cosina measure. For data filtering, we set Shannon Score threshold of 0.15 to retain at least 50% of annotated data.
http:///wrthivee..r//ddd///////s htttts:://openai.com/rrrte

---

Ubuntu
NYC
BC3
R-1
R-2
R-L.
R-1
R-2
R-L
R-1
R-2
R-L
ChatGPT (zero-shot)
39.9
17.4
29.1
40.7
14.5
26.2
43.6
14.5
27.0
ChatGPT (one-shot, random)
44.6
21.2
31.9
43.3
16.0
27.5
43.3
15.2
28.3
ChatGPT (two-shot, random)
47.3
24.4
36.0
42.5
16.4
27.9
44.8
16.2
28.5
ChatGPT (two-shot, USE-cosine)
49.0
25.9
37.2
42,9
16.6
28.3
46.9
16.0
29.9
PEGASUS (Zhang et al., 2020)
27.9
17.0
23.0
32.0
10.6
19.9
27.8
7.8
16.6
BRIO (Liu al., 2022b)
33.4
17.6
24.0
35.0
12.0
21.0
32.3
10.8
20.0
Standard KD
48.8
26.8
37.5
41.5
17.3
27.7
44.7
17.3
29.9
SPF KD
55.4
33.4
43.5
41.7
18.2
28.3
47.2
19.8
32.3
Table Performance of our proposed prlect-rrompt-fiteer (SPF) KD and baselines.
3.2
Model
Results and Analysis
In Table I, report the results of the proposed SPF-KD approach compared to standard KD, both using a pre-trained BART model fine-tuned on annotations generated by ChatGPT. Our baseline, standard KD, outperforms PEGASUS (Zhang et al., 2020), and BRIO (Liu et al., 2022b) showing the efficacy of KD and the importance of adding Stack- Exchange data. Compared standard KD, SPF KD achieves substantial gains of 6.6, 0.9, and 2.5 ROUGE-2 scores on Ubuntu, NYC, and BC3, respectively, showing the strength our proposed approach. The larger magnitude of improvement on Ubuntu can be attributed to a large proportion of similar forums in our Stack dataset that our data selection step may have over-represented. Similar to Wang et al. (2021). also observe that KD using ChatGPT summaries outperforms the corresponding ChatGPT teacher model itself. We attribute this to fine-tuning the smaller model on conversational (forums) data and being specialized to do only the summarization task. We provide a few examples of the outputs of the models in Ap- pendix A.2. This makes KD, especially our SPF KD, an effective and cheap strategy to build spe- cialized summarization models without relying external services.
For both Standard and SPF KD, we fine-tune one
student model each with the corresponding dis- tillation data and evaluate on all three sets.
Similar the findings of Demeter et al. (2023), our preliminary investigations indicated that fine- tuned BART model produces qutaer-quality sum- maries compared to fine-tuning models such as PEGASUS (Zhang et al., 2020) Hence, we use XSum-finetuned (Narayan et al., 2018) BART- large* (406.3M params, Lewis al. 2020) as the student model for both KD approaches. For each of our KD experiments, we further fine-tune this BART model on the corresponding distilled dataset for 5 epochs, beyond which we observed overfit- ting on our validation set. We used a batch size 16 and learning rate 2e-5.
We report results using various prompting strate- gies for ChatGPT: with no priming examples (zero- shot), with example(s) from the validation (one or two-shot, random), and with validation examples closest the test conversation based on USE-cosine mea- sure (two-shot, USE-cosine). ChatGPT (two-shot, random) is the teacher model for Standard KD due to the randomly selected prompting examples, and ChatGPT (two-shot, USE-cosine) is the teacher model in SPF KD as it incorporates the prompt retrieval step. We also include results of two com- petitive off-the-shelf summarization models: PE- GASUS (570.8M params, Zhang et 2020) and BRIO7 (406.3M params, Liu al. 2022b).
two reneoed priming
4.1 Ablation study
We tease apart the contribution of each step of our proposed approach to the overall performance improvement over the standard KD approach (see Table 2). We find that each of the three steps improves on the three datasets. Moderate gains are observed in NYC on adding each step. The data selection step notably improves BC3. Data filtering consistently improves performance across the three test sets. The biggest improvements are
Shetp::/if..cofaccbokklarr/ http://:///...ccoo/gll//////////
'https://hf. ct/ wwwpoooool////ggsuss.esssssspsee large http:////hhf..o/aae-Liiee
htt.://.ww...llllloooooooeeeennr uncased

---

Ubuntu
NYC
BC3
Ubuntu, 0.4 for NYC, and 0.5 for BC3. This shows we get more similar training examples for Ubuntu compared to the other two domains. Interestingly. the final ROUGE-2 gains using our SPF-KD ap- proach also follow the same ranking as this ag- gregated USE-cosine measure (Ubuntu > BC3 > NYC).
Standard KD
26.8
17.3
17.3
+
Data Selection
28.5
17.4
19.7
Prompt Retrieval
32.4
17.8
19.7
Data Filtering
33.4
18.2
19.8
Table 2: ROUGE-2 results of using each step in the select-prompt-filee KD approach.
5
Conclusion
25
We propose methodical knowledge distillation approach that finetunes a smaller LM for summa- rizing conversations with synthetic summaries gen- erated by ChatGPT. Our approach selects a suitable subset to be distilled, retrieves matching prompts for ChatGPT, and filters the ChatGPT-ggenerate summaries to yield better-quality training data. Our proposed method substuntially improves over standard distillation approach on three test cov- ering forum and email conversations.
ROUGE22
22
23
Standard KD SPF
18
25%6
50%
75%
100%
training Data
Figure 3: Average ROUGE-2 scores vs. percentage of distillation data used
Limitations
The limitations of our paper are:
observed in the Ubuntu test set at each step. We attribute this to the abundance of Ubuntu-related forums in StackExchange which are adequately exploited in our approach.
Our test sets relatively small, with only 120 samples. However, this limitation is due the lack of data in the conversational sce- nario.
4.2 Distillation Cost
Our experiments have been run using 100K training points while having access to 7 mil- lion unlabelled data points.
We study the performance of the standard KD (Baseline) and our SPF KD approach under lower ChalGPT-annotation budgets. To do this, we train the student model using randomly sampled subsets of varying sizes of the distillation data obtained using both Standard KD and SPF KD (after filter- ing) and compute the average ROUGE-2 across the three test sets. This is plotted in Figure 3. We find that fine-tuning the student model with 50% of SPF KD training data achieves similar ROUGE--2 score compared utilizing the entire Standard KD training data.
We do not explore a wide variety of prompting strategies.
We only validate our data-centric approach on the BART-large model while it can be applied to any large pretrained language model.
We only use metrics from the ROUGE family to evaluate models in our experiments.
4.3 Domain Relevance
References
We can use the aggregated USE-cosine metrics to estimate the relevance of the selected training data for each domain represented by each valida- tion set. For every validation example in the three domains (Ubuntu, NYC, and BC3), we compute the mean USE-cosine across the 833 correspond- ing selected training examples. We then average this value across all validation examples in the cor- responding domain. We find that we get 0.7 for
Emily M. Bender, Timnit Gebru, Angelina McMillan- Major, and Shmargaret Shmitchell. 2021. On the dangers of stochastic parrots: Can language models be big? In Proceedings the 202/ ACM Confer- ence on Fairness, Accountability, and Transparency.
Sumit Bhatia, Prakhar Biyani, and Prasenjit Mitra. 2014. Summarizing online forum discussions can dialog acts of individual messages help? In Proceedings of the 2014 Conference on Empirical Methods Natural Language Processing (EMNLP).

---

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems.
Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. 2022a. What makes good in-context examples for GPT-3? In Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures
Yixin Liu, Pengfei Liu, Dragomir Radev, and Graham Neubig. 2022b. BRIO: Bringing order to abstractive summarization. In Proceedings of the 60th Anmual Meeting of the Association for Computational Lin- guisties (Volume 1: Long Pupers).
Daniel Cer, Yinfei Yang. Sheng yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St. John, Noah Constant, Mario Suajardo-Cespedees Steve Yuan, Chris Tar, Yun-Hsuan Sung, Brian Strope, and Ray Kurzweil. 2018. Universal sentence encoder. arXiv preprint arXiv:1803.11173
Yongtai Liu, Joshua Maynez, Gonçalo Simões, and Shashi Narayan. 2022c. Data augmentation for low- resource dialogue summarization, In Findings of the Association for Computational Linguistics: NAACL 2022.
Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018. Don't give me the details, just the summary! topic-aware convolutional neural networks for ex- treme summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Lan- guage Processing. Association for Computational Linguistics.
David Demeter, Oshin Agarwal, Simon Ben Igeri, Marko Sterbentz, Neil Molino, John M. Conroy, and Ani Nenkova. 2023. Summarization from leaderboards to practice: Choosing a representation backbone and ensuring robustness. arXiv Preprint arxiv:2306. 10555.
Bosheng Ding, Chengwei Qin, Linlin Liu, Lidong Bing, Shafiq Joty. and Boyang Li. 2022. Is GPT-3 a good data annotator? arXiv Preprint arxiv:2212. 10450.
Colin Raffel, Noam Shazeer, Adam Roberts. Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text trans- former. The Journal of Machine Learning Research, 21(5)55485-5551
Nicholas Egan, Oleg Vasilyev, and John Bohannon. 2022. Play the Shannon game with language mod- els: A human-fres approach to summary evaluation. In Proceedings of the AAAI Conference Artificial Intelligence.
Jan Uirich, Gabriel Murray, and Giuseppe Carenini. 2008. A publicly available annotated corpus for su- pervised email summarization. In Proceedings of the 23th AAAI Conference on Articcial Intelligence in Enhanced Messaging Workshop.
Geoffrey Hinton, Oriol Vinyals, and Jeffrey Dean, 2015, Distilling the knowledge in a neural network. In NIPS Deep Learning and Representation Learning Workshop.
Shuohang Wang. Yang Liu, Yichong Xu, Chenguang Zhu, and Michael Zeng. 2021. Want to reduce la- beling cost? GPT-3 can help. Findings of the Association for Computational Linguistics: EMNLP 2021.
Mohit Iyyer, Varun Manjunatha, Jordan Boyd-Graher, and Hal Daumé III. 2015. Deep unordered compo- sition rivals syntactic methods for text classification. In Proceedings of the 53rd Annual Meeting of the As- sociation for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers).
Carole-Jean Wu, Ramya Raghavendra, Udit Gupta, Bilge Acun, Newsha Ardalani, Kiwan Maeng. Glo- ria Chang. Fiona Aga, Jinshi Huang, Charles Bai. et al. 2022. Sustainable Al: Environmental implica- tions, challenges and opportunitics. Proceedings of Machine Learning and Systems.
Yoon Kim and Alexander M. Rush. 2016. Sequence- level knowledge distillation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing.
Jingqing Zhang, Yao Zhao, Mohammad Salch, and Peter Liu. 2020. PEGASUS: Pre-training with extracted gap-sentences for abstractive summarization. In In- ternational Conference on Machine Learning, pages 11328-11339.
Mike Lewis Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. pre-training
BART: Denoising requence-to-sequee for natural language generation, translation, and com- prehension. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics.
A
Appendix
Pengfei Li, Jianyi Yang, Mohammad A. Islam, and Shaolei Ren. 2023. Making Al less "thirsty": Uncov- ering and addressing the secret water footprint of AI models. arXiv Preprint mrxiv:2304.0322..
A.1
Prompt Structure
The format of the prompt given to ChatGPT is given below:

---

one example of sumariiaatoon
Conversation: cow.eessantnestiinonnn
Summary:
<example_sueee summurryyyyy
Please summarize the following conversation: <conversation>
We pass prompt in the above format part of the user's input to ChatGPT get response from it.
A.2 System Outputs
In Table and Table we show output summaries generated by ChatGPT (two-shot, USE-cosinc), Standard KD, and SPF KD.

---

Conversation
Summary
<userA>: : I'm driving from Cleveland. I want to stay dowentown and see a few sights on Saturday, to the game on Sunday (unless there's a shuttlss), then head to the casino Niagara Falls. How far is Seneca from dowatown Butfalo? Are there any other casinos? How for Niagara Halls? Can see enough from US side, should suck up and get a and go into Canada? Any recommendations would he apprecisted.
ChatGPT: The comversation is about person who is plansing to visit Ruffalo to attend Sabres game. They were looking for recommendations for restaurants near their Participants made several recomncndaiio,, including Sakariit,, Papaya. Buffalo Chop House, and F.B. Green's s. They also usked for advice the free train that could lake them to the arenn and back to their hotel. They were also interested in visiting the casino or Niagara Falls. There was discussion about the Canadian Casinos versus Seneca casino on the US side. They also wanted to know about tail gating and if alcohol permiited in the parking lot.
<userB> : You
pick up special Game Day bus to Ralph Wilson Studium: htpp//wwwwn nfta con/mnttrrygmmccayybbl asp Niagara Falls about 20 minutes from down- town Buffalo. You can enjoy the falls from the U.S. side, but if you want to see the you'll
Canadian falls, you won't need passport
is a hirth certificate and govemmennt-ssoudd photo ID. I'll let someome chee answer your casino questions.
<userA>: Thanks. Can you recommend sa we can hop the metro to go to the game? The gume is November 17 what weather should I expeet?
that close the
station downtown
Standard KD: The conversation is about person who wants to go to Buffalo a Ruffalo Rilks game and wants know how far it is from downtown Buffalo, if there are any other they can enough
<userls: You expect the same weather Butfalo you get Cleveland. get your weuther probubly the day after you get it (from the wesl). Il could ruin it could snow, it could
sunny day... who knows Regarding botels downtown. here's is list from a previous post: 1) Hampton Ias tripadvisor. comfoooll Soviow-66094--224990 92.4999........ 2) Adam's 's Mark ripadvisor.comHont
casinos, how far Niagara Falls, from the US side or if they should get passport and go into Canada. suserBo recammends a special Game Day bus to Ralph Wilson Stadium and Ningara Falls. <uzerD>> recommends hotels downtown. including Hampton Inn. Adam's Mark. Hyatt Regency Hotel, and Comfort Suites. The Seneca Niggara Casino is within walking distance of uctual Falls and there casino in downlown Bullalo. <user P> advises against going to Canadian casinos as they tight and do not offer drinks. <userG> recommends checking out both Canadian and Seneea casinos. <userd> thanks everyone for their advice and decides to book the Country Ian near the Falls Canada for three nights. They will drive back on Monday and check in to the hotel in Hamburg and then go to the stadium early for tail gailgainn.
Review- (69974499999 I...... Hyant Regency Hotel tipadvi- -60974--9911222 R... Comfort Suites coo Review- 26097440931 1114449912--44 28--... You can eatch the train on Main Street and it's free as long as you stay ahove ground. The station where you will most likely catch the shuttle
son.omllleee Review- Reviele
game near Main and N. Division St. across from Church St. Nigara Falls: The Seneca Nisgara Casino within walking distance the actual Falls,
a mise day and you leel like walking, it's an easy walk to the Canadian side acroes the Rainhow Bridge. Just bring you birth ccrt and driver's is license they let you back into the USA. There is casino in downtown Bílo, but it's nothing special, from what I've been told.
<userRa: : the metro main does not our to ralph wilson stadium. the stadium in orchard 90 you drive uplif you drive the leave your hotel early as traffic can got thick. consider driving, tailaating is art in buffale! it's lot of fun experience. would recommend the hampton inn. good central location for night life.
park, about 25 min south buffalo. (you'll 'll puss
<userf>: Don't i waste your time money at the Canadian casinos. The slots are VERY right and you will neser see a minimum below 15/205. Also, If you're 'o a smoker, they are non smoking and do NOT offer free drinks, whereas the Seneca casino on the US side does. Go Bills! Boe Brewns!
SPF KD: person driving from Cleveland Buffalo and wants to see few sights on Saturday, drive to the game on Sunday, then bead to the casino or Ningara Falls. The person wants to know how far Sencen is from downtown Buffalo, there any other casinos, and they enough from the US side, or should they get a passport and go into Canada. A reply suggests that the person can pick up a special Game Day bus to Ralph Wilson Stadium, and Nisgara Fulls is about 20 minutes from downtown Butfalo. The reply also recommends hotel to the metro station downtown, and the weather in Buffalo is the sume as Cleveland. The Seneca Niugura Cusino is within walking distance of the actual Falls. If it'ss a nice day and the person foots like walking. it's an walk to Canadian side across the Rainbow Bridge. There casino in dowatown Buffalo but it's nothing special. Amother reply suggests leaving the hotel early trafh can thick. They recommend the Hampion Inn, which is a good central location for night life. The Canadian casinos are far superior to the Seneea if the person's is motivation is a nice stmoddphere. shopping, and heauty.
enserGs No offinse to Gear but the Canadian Casinos are far superior to the Sensca your motivation nice atmoshphere, shopping, and beauty. can not speak for the gambling outcomes. To the Seneca's is credit, they have great resiaurants and a very nice concert venue. If you are into casinos, I would check out both...
of your advice. We went ahead hooked the Country Inn near the Falls in Canada for 3 nights (I figure we can check out the Falls since the only time I was there when was toddler). We'll drive back Moaday
<userAs: Thanks
the hotel Hambury (Holiday Inn....heeap and then go over to the stadium early for some tuil milggatin.. looks like it's only about miles away, su big of a hendache. 1s alcohal permitted in the parking lot? Little gus grills? And whut about the restroom situation-tnn there port-o-johns available?? And how soon do they open the stadium? Thanks so much for your help!
check
close
with cruzy tralli I'm hoping its not lou
<ueerG>: drinks-yes, grill--yes the lots open at the crack of dawn....it is full contact tail- gaiting Ruffala and think after experiencing you change your team afti liation. After going to a few Browns I believe BuiTalo has the tailguiting title...ut it is very close....Go Toronto..oo oh I mean Bulfalo...!! !!"
Reference: A person was driving from Cleveland to Buffalo to attend the game Sunday, Further he planning go casino or Nagarra Fulls. He wanted to how far be Seneca. from downtown Buffalo, were there any other casinos, how far was Niagara Falls and was passport required to go into Canada? Someone replied that he can pick special Game Day bus to Ralph Wilson Stadium. Niagura Falls was about minutes from downtown Burfalo. To travel to Canadian side Ningara fall, birth certificate and a government issued photo ID will be needod. The other reply was that there were many horels in the downtown. The mains and buses were available transporatiion. The Seneca Niagara Casino was within walking distance of the actual fall, The other suggestion was waste time money the Canadian casinos. There were some disagreements over the quality of the Canadian casinos.
Table 3: Summaries generated by ChatGPT (two-shot, USE-cosine), Standard KD, and SPF KD on an example
NYC forum conversation (usernames are anonymized).

---

Conversation
Summary
Since yestenday (when I installed the latest Ubuntu 'odgy' kemel security update), my Serval Performance system failed
<userA::
The user's s Serval Performance system failed
ChatGPT:
soveral times. This problem required hard shuidown rebool. The first time this occumed, I tred booting into recovery mode, which secmed to work arcund the issue. The second time, even recovery mode did work; three times attermpinn boot wichout Since could sce the boot status this time noticed the message" "BUG: soft lockup on CPUAD" 'This did not sound very good to me. Fortunately. have just hooted my system successfilly again after previously turned since I was not using it the time and had noticed an appurent CPU bug whereby one of my vores was system monitor. Anyhow. don't know if the switch was the cause/cure or not, hecause when the problem first began still managed to boot with the switch "off" and I have only had sucvess with it 'on'. need de some important work before can test this theory and risk not being abic boot again.) ) Has anyone clse heard of or experienced a similar problem? -cuerts-
several times after installing latest Uhunma 'odgy" kerne sccurity update. The noticed the message "BUG; soft lockup CPUAO", The user tried booting into rocovery mode, which worked the first time but noe the second time. The user turned switch, and the problem stopped occurring. The user noticed that one CPU 100% frequency when coming out of hibernation. The reinställd the nvidia gix package. The problem secmed to be related hibernation. Another uver suggested that the problem could be caused by missing
flipping wireless card switch on the chascis the 'on' position.
wireless card
always 100% frequency
according
always at
cuserBo: Hi cuserto have one other report "BUG soft lockup CPENO" although was different machine. I'm trying to re-crvate the problem on Serval now and will get buck to you shortly. <user(s: Perhops
fuuf/ibbxorg/moode
caused by this? http:///wentufoommss org/showthreed, phpp (tm $18206 <userBs; <aserd>- we're re not experiencing the problem using a Vidiaa's is driver from the repos does hackmasters thread look relevant?
Standard KD: cusends's Serval Performance system has failed to bool several times since installing the latest Ubuntu "edgy" kereel security update. The problem required a hard shurdown and reboot. The first time, <usert> tried booting into recovery mode, which seemed to work around issuc. The second time, even recovery mode did not work; three times cuueedo artempted boot without success. <mvert> noticed the message "BUG soft lockup on CPURY" and riipped the wireless card switch on chassis to 'on" position. However, be is umsue if the switch was the causelcure or mot. <wsert>>> and <merrC> suggested that the problem may have been eansed by CPU bug where one of the 100% frequency even when not use according to the system monitor, The ismu scoms to he related coming out of hibernation. <usert>> system fresh, he does not observe the behavior, but if hibernates and resumes, he notices
<aserAAc do have the driver version mentionad in that norice. However, neither the files nated ure missing from lus/lia/sorr/mmnaaless Also, my bool process was failing well before attempking to initialize X. now reinstalled the midia glx package just in case. Since flipping wireless back to the 'on' position. I have not had this problem. I haven't I yet tried switching it 'otl' again to see Regarding the istue of epu nunning at 100% frequency all the time, my best guess now is that il may not be reluted to the wireless card/driver. I have recenily noticed that the issue seems to he related to coming out of hibernation, It boot system fresh, do not observe the behavior, I hibernate
swisch
if reproduce
resume then notice Many thankss to help. If you huppen to discover any more information about the ettor message I initially posted, let me know. Otherwise. have my workaround for now. -wwerd
hackmeister your
always at
SPF KD; <usert>> installed the latest Ubuntu "odgy" kernel security update and the Serval Performance system failed to boot several times. The problem required hard shutdown and rebucs. The user tried heoting into recovery mode, which seemed to work around the issue, but the second time it did not work. and the user naticed the message "BUG: soft lockup CPU4D' The user has just booted the system snacessfilly again after fiipiig the wireless card switch on the chassis to the 'on" position. However, if the switch was the canselcure or not. The issue be related to coming out of hibernation. and if the user bools the system fresh, they do not observe the behavior, but if they hibernate and resume, they notice it. cusento has reinstalled the avidis-gls puckage just in case.
user
Reference: The user's serval performance system has Railed to boot several times. The problem required a hard shuldown and reboot. When the prohlem occurred first time, the user tried booting into recovery mode, which seemed work around an issuc When the problem occurred second time. recovery made did work. The tried
booting the system after Ilpppnng the wireless card switch on the chassis to the 'on' position. The user noticed the message BUG: soft lockup on CPUAO. The user also noticed that one of the cores was always al 100% frequency. even when according to the system monitor due to the CPU bug. The other user found the repoet which had mentioned problem due to the same bug and was trying to recreate the problem on serval. The cause the problem might mentioned this webpuge. http://ubuntufooum frrunttf org/show wrg/showhreaad php2t=3 16206 The user had the driver version information. The user felt that the wireless card/driver was not related the CPU frequency issue but il related tn the system coming out hibemaation. that the system was booted the problem was not observed when the system was resumned from libernation, the problem was observed.
user found
Table 4: Summaries generated by ChatGPT (two-shot, USE-cosine), Standard KD, and SPF KD on an example
Ubuntu forum conversation (usernames are anonymized).


Reviews:
[{"review_id": "lQOUdK7nM6", "reviewer": "Reviewer_8qtH", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer 8qtH,\n\nWe thank you for your valuable feedback on our paper. Please find our responses below.\n\n> **_“Limited to forum conversations, SPF KD's applicability to diverse text types might be uncertain.”_**\n\nAutomatic abstractive summarization carries significant value for the conversational domain given the massive usage of online communication channels (forums, chats, emails etc.). We lack large-scale annotated conversational corpora for training summarization models. For this reason (and due to ChatGPT usage costs), we focused our proposed approach and experiments on conversation summarization (mentioned in the Limitations section).  However, it is important to note that we have evaluated our approach on two text types (forums and emails) and two domains (IT and travel).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. SPF KD offers a novel way to optimize LLMs for low-resource tasks, showing strong summarization improvements.\n2. Addresses expensive LLM fine-tuning, providing a cost-effective solution for NLP tasks.", "reasons_to_reject": "Limited to forum conversations, SPF KD's applicability to diverse text types might be uncertain.\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "\nThis paper presents a novel approach, \"select-prompt-filter knowledge distillation\" (SPF KD), aimed at efficiently using large language models (LLMs) for summarizing conversations in low-resource domains. The challenge addressed is the costly fine-tuning of LLMs on specialized tasks. SPF KD involves selecting conversation data similar to a small annotated set, generating prompts for LLMs, and filtering low-quality data. The approach shows significant performance gains in summarization tasks, making it valuable for tasks in resource-scarce domains. Overall, the paper introduces a strategic solution for optimizing LLMs' utility in summarization while minimizing costs.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "mOlVmUP3Ti", "reviewer": "Reviewer_jSRx", "summary": "", "questions": "1. How does the diversity of prompts affects the quality of the distilled models?", "limitations": "", "rating": 2, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer jSRx, \n\nWe thank you for your valuable feedback on our paper. Please find our responses below.\n> **_“Producing training examples from larger models and using them for training smaller ones is known to be effective for many NLP tasks, which limits the novelty of this work.”_**\n\nOur paper presents a novel pipeline to refine and improve the standard distillation process (which is our baseline). We do not claim that the novelty of our approach lies in directly applying knowledge distillation. Specifically, our paper focuses on the sequential application of three steps in the LLM distillation process that improves summarization performance (see Section 4 of the results) and minimizes the costs of generating the data (see Figure 3) over the standard distillation process.  \n\n> **_“How does the diversity of prompts affect the quality of the distilled models?”_**\n\nWe explored using 1-shot and 2-shot prompts and found that the performance of ChatGPT increases with increasing the number of priming examples. This results in an improvement in performance of the 2-shot over the 1-shot scenario by +1/+1.5/+1.5 ROUGE-1/ROUGE-2/ROUGE-L scores. We conducted the distillation experiments with 2-shot prompts for ChatGPT assuming a better teacher model results in better training data for the distilled models.\n\n> **_“$D_t$ is never introduced formally”_**\n\n$D_t$ has been introduced in line 151 as a notation for the unannotated training dataset. We will state that it is a set of forum threads.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The proposed approach improves the results over the teacher model ChatGPT using a substantially smaller model (i.e., BART).\n- More automatically produced data leads to higher results, and the model outperforms a standard finetuning baseline on data from an external task.", "reasons_to_reject": "- Producing training examples from larger models and using them for training smaller ones is known to be effective for many NLP tasks, which limits the novelty of this work. \n", "questions_to_authors": "1. How does the diversity of prompts affects the quality of the distilled models?", "paper_topic_and_main_contributions": "The paper presents an approach for distilling conversation summarization capabilities using the outputs from a large language model (i.e., ChatGPT). The authors propose a three step approach based on: 1/ example (conversation) selection from a set of unlabeled ones, 2/ prompt retrieval (matching the closest prompt to the conversation), and 3/ automatic summary generation and filtering out the inconsistent outputs. The approach is evaluated on three different summarization datasets (Ubuntu, NYC, BC3).", "excitement": 2, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "- Dt is never introduced formally"}, "extra_text": {"Missing_References": "1. Xu, Can, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. \"Wizardlm: Empowering large language models to follow complex instructions.\" arXiv preprint arXiv:2304.12244 (2023)."}}, {"review_id": "17HiDjF0AT", "reviewer": "Reviewer_AQP7", "summary": "", "questions": "Question A: When comparing the performance improvements across the three datasets, you notice that the Ubuntu dataset shows the biggest gains. Can we estimate if the data from an unannotated corpus would be a good fit for a given validation set? Is there a significant difference in the average cosine similarities between the closest samples from the unannotated corpus and each of the validation datasets in the selection and prompt phases?\nQuestion B: What embeddings are you using to compute the USE cosine similarity - GloVe?\nQuestion C: Do you have any insights on the quality and factual correctness of the generated summaries?\n", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer AQP7, \n\nWe thank you for your valuable feedback on our paper. Please find our responses below.\n\n> **_“The paper uses XSum BART for its experiments but it doesn’t specify why this model was selected.”_**\n\nWe had tried several pre-trained models (including PEGASUS, T5) in many of our initial explorations with fine-tuning and knowledge distillation for summarization. We found that fine-tuned BART generally performs best and produces good output summaries. A very recent preprint also reports a similar finding (Demeter et al, 2023, https://arxiv.org/pdf/2306.10555.pdf) that shows fine-tuned BART produces better outputs compared to PEGASUS and T5, both in terms of human and automatic evaluation. We will add details regarding our choice and this citation in the revised paper.\n\n> **_“Also, there is information about the training parameters but not how they were selected.”_**\n\nThe training parameters were either experimentally set based on validation performance or due to experimental constraints. For instance, we observed overfitting of our models after 5 epochs of finetuning and 100K training examples were chosen due to our ChatGPT annotation budget. We will add more details in the revised paper.\n\n**Answers to questions:**\n\n> **_Question A: “Can we estimate if the data from an unannotated corpus would be a good fit for a given validation set?”_**\n\nYes, average cosine of each training example across the validation examples can indicate if the unannotated dataset is good for it. For example, we found that Ubuntu, NYC, and BC3 dev sets had an average cosine of 0.7, 0.4, and 0.5, respectively across the selected training examples. Our improvements on these test sets with our selection method also follows the same order (Ubuntu > BC3 > NYC). \n\n> **_Question B: “What embeddings are you using to compute the USE cosine similarity - GloVe?”_**\n\nWe use Universal Sentence Embeddings (Cer et al. 2018, https://tfhub.dev/google/universal-sentence-encoder/4) as the sentence representations. We give details about the same in Line 121-126 in the paper (Section 2, ‘USE-Cosine’)\t\n\n> **_Question C: \"Do you have any insights on the quality and factual correctness of the generated summaries?”_**\n \nWe inspected some of the conversations and the generated summaries of our system and ChatGPT. Overall, the systems are mostly on par in terms of factual correctness. However, in one example, we noticed that ChatGPT hallucinated information that was not entailed from the input conversation (The conversation is about a person who is planning to visit Buffalo for a football game. However, ChatGPT hallucinated that the conversation was about going to  a game by “Sabres” which is a hockey team from Buffalo). Compared to our system that is tuned for the summarization task, ChatGPT has more generative capabilities given the data it has seen during training. This makes it capable of answering outside-of-context potentially leading to factual inaccuracies. We also noticed that ChatGPT tends to produce shorter summaries than our system leading to the omission of important facts from the summary for some of the conversations.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "•\tA well-structured and well-written paper.\n●\tThe paper presents an approach for generating weakly supervised datasets for summarization which can be used for more efficient model training and can make up for the lack of training data\n●\tThe authors show another way of leveraging LLMs to distill knowledge and generate annotated datasets\n●\tThe approach can be applied in different domains\n●\tThe approach shows improved results over direct LLM usage and competitive models\n", "reasons_to_reject": "●\tThe paper uses XSum BART for its experiments but it doesn’t specify why this model was selected. Also, there is information about the training parameters but not how they were selected.", "questions_to_authors": "Question A: When comparing the performance improvements across the three datasets, you notice that the Ubuntu dataset shows the biggest gains. Can we estimate if the data from an unannotated corpus would be a good fit for a given validation set? Is there a significant difference in the average cosine similarities between the closest samples from the unannotated corpus and each of the validation datasets in the selection and prompt phases?\nQuestion B: What embeddings are you using to compute the USE cosine similarity - GloVe?\nQuestion C: Do you have any insights on the quality and factual correctness of the generated summaries?\n", "paper_topic_and_main_contributions": "The paper presents an approach for generating weakly supervised annotated datasets for data summarization using knowledge distillation of LLMs (ChatGPT). It addresses the problem of scarcity of annotated training data for finetuning summarization models for a specific domain. The main contributions of the paper include an approach for data efficiency as well as an approach for generating new language training resources.\nThe proposed approach consists of three steps - select, prompt, and filter, which generate relevant training examples from a big unannotated corpus. The selection phase identifies the most similar instances from the unannotated corpus to the validation examples using cosine similarity of sentence embeddings (Universal Sentence Encoder). Once the examples are selected, the prompt phase generates a few-shot prompt to generate the summary and uses ChatGPT to obtain the summary of the example. In the filtering phase, the synthetic examples that do not satisfy a certain Shannon Score threshold are removed from the dataset.\nThe paper performs experiments using 3 different forum conversation datasets in the general domain (Ubuntu, NYC, BC3). Using the proposed approach, a distantly supervised dataset is generated and used to train a language model (XSum BART). The authors compare the ROUGE-1/2/L metrics of BART trained using the generated dataset with several other systems - ChatGPT (zero/few-shot), BRIO, and Pegasus. Using the generated dataset, the fine-tuned model shows improved performance over all 3 datasets and metrics.\nThe authors perform an ablation study for each part of the process - select, prompt, and filter which shows that each part contributes to the improved performance but the improvement varies based on the test set.\nThe paper presents a practical approach that can be applied to different summarization datasets and domains and can help tackle the problem of insufficient training data. Also, it efficiently uses the resources to generate summaries only for relevant samples from the unannotated dataset. Experiments show using the proposed approach as opposed to a random selection can achieve the same ROUGE-2 score using just half of the number of training examples, which reduces the cost and is more environmentally sustainable.\n", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "The paper is well-written and easy to follow.\nMinor suggestions:\nIn Table 1 it would be helpful if you distinguish between the dataset and model used in the experiment as well as create separate subcolumns for ROUGE-1/2/L.\nIf space allows, showing an example of a text and summary would be helpful.\nIt would be very valuable if the authors could publish the code and the generated dataset.\n"}}]

Author rebuttal:

Meta-review: The paper brings merits regarding "select-prompt-filter knowledge distillation" and enhance the summarization for low-resource domains, which is valuable. The paper is expected to add more tech details as mentioned in the rebuttal.|The paper presents an approach for generating weakly supervised annotated datasets for conversation summarization using the outputs from a large language model ; showing strong summarization improvements for low-resource settings. The reviewers agree the paper is sound but have concerns about the incremental nature of the approach and the limited applicability.
Decision: Accept-Main