InheritSumm: A General, Versatile and Compact Summarizer by Distilling from GPT
Authors: Yichong Xu
Ruochen Xu
Dan Iter
Yang Liu
Shuohang Wang
Chenguang Zhu
Michael Zeng
Venue: EMNLP 2023 Findings (2023)
Keywords: summarization
distillation
zero-shot
few-shot
large language model
TL;DR: We distill a versatile and compact summarization model from GPT-3.5 and achieve similar or superior performance in zeroshot and fewshot settings.

Abstract: While large models such as GPT-3 demonstrate exceptional performance in zeroshot and fewshot summarization tasks, their extensive serving and fine-tuning costs hinder their utilization in various applications. Conversely, previous studies have found that although automatic metrics tend to favor smaller fine-tuned models, the quality of the summaries they generate is inferior to that of larger models like GPT-3 when assessed by human evaluators. To address this issue, we propose InheritSumm, a versatile and compact summarization model derived from GPT-3.5 through distillation. InheritSumm not only exhibits comparable zeroshot and fewshot summarization capabilities to GPT-3.5 but is also sufficiently compact for fine-tuning purposes. Experimental results demonstrate that InheritSumm achieves similar or superior performance to GPT-3.5 in zeroshot and fewshot settings. Furthermore, it outperforms the previously established best small models in both prefix-tuning and full-data fine-tuning scenarios.

InheritSumm: A General, Versatile and Compact Summarizer by Distilling from GPT
Yichong Xu, Ruochen Xu, Dan Iter, Yang Liu, Shuohang Wang
Chenguang Zhu, Michael Zeng
Microsoft Azure AI (yicxu, ruxx, iterdan, yaliul0, shuowa, chezhu, nzeng) 8mi croosfttt.coom
Abstract
fore. LLMs are highly attractive for wide range of summarization applications.
While large models such GPT-3 demonstrate exceptional performance in zeroshot and few- shot summarization tasks, their extensive serv- ing and fine-tuning costs hinder their utiliza- tion in various applications. Conversely, previ- ous studies have found that although automatic metries tend to favor smaller fine-tuned mod- els, the quality of the summaries they generate is inferior that of larger models like GPT-3 when assessed by human evaluators. To address this issue, we propose INHERITSUMM, a versa- tile and compact summarization model derived from GPT-3.5 through distillation. INHERIT- SUMM not only exhibits comparable zeroshot and fewshot summarization capabilities to GPT- 3.5 but is also sufficiently compact for fine- tuning purposes. Experimental results demon- strate that INHERITSUMM achieves similar or superior performance to GPT-3.5 zeroshot and fewshot settings. Furthermore, it outper- forms the previously established best small models in both prefix-tuning and full-data fine- tuning scenarios.
Despite these remarkable achievements, training and deploying LLMs for summarization is com- putationally expensive. Deploying LLM for inference is already impractical for many NLP prac- titioners, especially for summarization where long input length of the document and demonstrative ex- amples are typical. Moreover, the prohibitive cost to finetune an LLM makes them hard to adapt to various custom domains, when abundant training data is available. This largely hurts the widespread adoption of LLMs for summarization. To this end, Zhu and Zeng (2022) proposes the target of achiev- ing all three aspects of the impossible triangle for NLP models: moderate model size, superior ze- roshot fewshot learning capability, and superior supervised learning capability.
In light of these challenges, we propose INHER- ITSUMM. The model exhibits a compact size and offers cost-effective fine-tuning and deployment capabilities. Its versatility is demonstrated by its applicability in both zero-shot/few-shot settings as well as supervised settings. Furthermore, the model's generalization capability is evidenced by its promising performance across wide range of benchmarks in all mentioned settings.
1 Introduction
Recently, the development of large language mod- els (LLMs)) like GPT-3 (Brown et al., 2020b) and PaLM (Chowdhery et al.. 2022) has largely revolu- tionized the text summarization community, bring- ing a new paradigm in the the way summaries are generated. LLMs have demonstrated unparalleled ability to produce highly readable summaries while requiring little to no training data, overwhelmingly prefered by human annotators than those from smaller models (Goyal et al., 2022). Human evalu- ators note that GPT-3 generates summaries with su- perior readability and coherence, sometimes even more favored than human-generated summaries (Liang al., 2022). One of the key advantages of LLMs is their capacity for zeroshot and few- shot learning (Brown et al., 2020b), which enables them to adapt to new domains with ease. There-
INHERISUUM is trained using knowledge dis- tillation from the GPT-3 model, by mimicing the GPT-generated summaries on general docu- ments. To facilitate this, curated the GPT- SUMM dataset, with over million document- summary pairs. The documents are collected from general language corpora and GPT-3 generates the corresponding summary for them. Utilizing this dataset, we train a ZCode++ (He et al., 2022) model by following the GPT-generated summaries in both zeroshot and fewshot learning settings.
One important limitation of fewshot learning for summarization is the input length limit: The in-context demonstrations consume valuable input

---

space, making it challenging incorporate many in-context examples within the prompt. To address this issue, we propose utilizing succinct demonstra- tions that aim to shorten the demonstration docu- ments and subsequently allow for the inclusion of more in-context examples.
Datasets
Domain
Does (%)
Length
The Pile
General
5.3M
1,296
ArXiv
Academic
203k
6039
CNN/DM
News
287k
WikiHow
Instructions
230k
578
Table I: Sources of Documents in GPTSUMM.
By training on the knowledge in GPTSUMM and using succinet demonstrations, INHERITSUMM is able to closely match, or some- times even outperform GPT-3 in zeroshot and fewshot summarization. Therefore, our INHER- ITSUMM with only 398M parameters able to achieve the impossible triangle (Zhu and Zeng, 2022) for the summarization task. We show that IN- HERITSUMM has strong performance and most ver- satile summarization capability to date: it achieves best close-to-best performance in all the settings including zeroshot/fewshot learning with prompts, fewshot learning with prefix tuning, as well as fully supervised learning.
show that
illustrative examples, and / is the task description. For prefix tuning and supervised learning, P is empty and X = D. We tune parameters 0 to adapt the model to given task.
Figure describes our overall method. We dis- till summarization skills from GPT-3 by using generate summaries for documents from different domains, including general documents used for lan- guage modeling, as well as specialized documents from labeled datasets. The GPT-generated sum- maries and (relatively fewer) human generated sum- maries from labeled datasets forms our GPTSUMM dataset. We then use GPTSUMM train seq2seq model, and then adapt zeroshot/fewshot learn- ing, prefix tuning and supervised learning summa- rization.
Our key contribution are three-fold. Firstly, we build the GPTSUMM dataset with more than 4M maragraph-summase pairs by using querying GPT- 3.5. To the best of our knowledge, GPTSUMM is the largest corpus to date focusing on distilling from GPT-3.5, and the first one focusing on sum- marization. Secondly, we build INHERITSUR UMM based on the GPTSUMM corpus. We show that IN- HERITSUMM exhibits versatile capabilities across zeroshot, fewshot, and supervised settings, achiev- ing the impossible triangle for summarization (Zhu and Zeng. 2022). Lastly, we propose methods include in-context examples for fewshot summa- rization that improves the performance.
In the following subsections, we first describe the method to build the GPTSUMM dataset in Sec- tion 3, and then introduce the model training pro- in Section 4. Finally we describe our method to adapt the pretrained INHERITSUMM model zeroshot, fewshot, and supervised summarization.
3 Distillation Data Collection of GPTSUMM
We discuss the construction of GPTSUMM the data for distillation in this section.
2 Problem Statement and Overview
3.1 Document Collection
In summarization, the goal is to summarize the input document D into short summary Y. For ze- roshot and fewshot learning, we can add a prompt P to the document to familiarize the model (param- eterized with 0) with the given task. A sequence-to- sequence model then takes the input X D] and predicts the summary Y
To increase the generalization of downstream mod- els, collect corpus of documents from var- ious sources. We first include 3.1M documents from the Pile corpus (Gao et al., 2020) We filter out non-English documents or the ones with too many non-character symbols 2, We truncate each document to be within 4096 words and remove du- plicates following (Smith et al., 2022). To include document from diverse domains and get closer to downstream tasks, we also include documents from arXiv(Cohan al., 2018), CNN/Daily Mail(See et al., 2017) and WikiHow (Kuuaee and Wang.
In zeroshot learning, P is a short description of the task, e.g., "summarize the following document into a short paragraph". For fewshot learning with in-context examples, the prompt consists of several descriptive examples along with an instruction, i.c.,
P = [X:; ..., Xn, Yn; IJ, where X, and Y,
'Our method also applies to the case where the prompt content appears after the document D)
"Namely, we remove documents where the percentage of non-English character symbols is larger than 70%.

---

Fewshot/Zeroshot
Zeroshot
GPT-3
Zerohhot
Inference
Fewshot
General Docs
Finetune
Prefix Tuning
GPT Summaries
Small LM
Finetune
Fewshot
Fully Supervised
Supervised
Docs/Sums
Human Summaries
Figure I: Overview of our method.
Index
ICD Docs
ICD Summaries
ICD Num
Input Does
Target Summaries
Quantity
None
None
General
GPT (zeroshot)
0.5M
General
GPT (from
General
GPT (fewshot)
2.6M
Supervised
Supervised
General
GPT (fewshot)
2.2M
None
None
0
Supervised
GPT (zeroshot)
0.6M
Supervised
Supervised
up 4
Supervised
GPT (fewshot)
0.5M
Supervised
Supervised
up 4
Supervised
Supervised
0.6M
Table 2: Modes of generation in GPTSUMM. ICD means in-context demonstrations. General means documents from the Pile corpus. Supervised corresponds the labeled datasets Table 1. The quantity listed are after the filtering steps specified in Sec 3.
2018) datasets. Table describes the composition documents in detail.
ate summaries for an additional 2.6M documents in the Pile, which we denote as data type 2 In ad- dition to the zeroshot examples from GPT, we also leverage document-summarry pairs from supervised datasets (Table 1) as demonstrations to obtain data type 3
3.2 Summary Generation
We utilize the GPT-3.5 model, specifically the text-davincis 002 variant, to generate sum- maries. To adapt to downstream use cases, we ap- ply different inputs, prompts, number of in-context demonstrations and zeroshot / fewshot settings to GPT to continue improving the quality of generated data, as shown in Table 2.
For parts 4516, we use the supervised datasets as the input documents. GPT to generate summaries in either zeroshot or fewshot settings, using the supervised datasets as in-context demonstrations. Finally, in 6, we employ the supervised datasets themselves to learn in multitask learning fashion. We use a specific "following" instruction (detailed in Appendix A) to follow the in-context demonstrations for part 6 enable the model produce diverse outputs com- pared to 3 Through this approach, our model can follow the given instructions to generate distinct summaries for the same documents.
and (5) we utilize
Initially, we collect instructions from the Prompt- Source dataset (Bach et al., 2022) and filter out those that only produce a "subject line" rather than a complete summary. This process yields final set of 24 instructions. To generate examples for few- these instructions to produce
shot learning, summaries for 500k documents in the Pile corpus (referred as "General") in zeroshot manner, re- sulting in data type ①. After removing summaries with low quality (as detailed in section 3.3), we use these summaries as in-context examples to gener-

---

3.3 Quality Assessment
ever, documents in the summarization task are typ- ically lengthy, leading reduced usable input length for GPT and distilled models when includ- ing numerous in-context examples. To alleviate this problem, we propose to truncate the document and summary to include more examples. To do this, we first truncate all training documents and summaries in the supervised datascts to length & (k 256 in our experiments). For every truncated docu- ment/summary, we add suffix "<omitted, words in total>" to the text, where / is the length of the original text. More specifically, for every document D longer than ho words, we construct a succinet truncated document D' = [d1, d2,..., .....,,, da. <omitted I I(D) words in total>]. where dj is the j-th word in D, and similarly for every summary S. We then add up M (M 4 in our experiments) succinct document-summary pairs in-context examples for any given input document, and ask the model (either GPT or distilled models) to fol- low the in-context demonstrations. We apply suc- cinct demonstrations to data (5) and (6) with ICDs from supervised datasets. We do not apply succinet demonstrations to 2 and (3 since we observe that altering the number of in-context examples does not significantly impact the quality of generated summaries for general documents from the Pile corpus.
To ensure the quality and relevance of the gener- ated summaries, estimate the summary quality and apply filtering process consisting of multiple stages. The first step involves retaining only those summaries with a "finish reason" of "stop." This indicates that the model has reached a natural stop- ping point, which serves as a reliable signal for the summary's 's quality.
Subsequently, the text undergoes post- processing to eliminate any superfiuous content. If the text contains a string that indicates another round of generation, such as consecutive newlines, the text is divided at the consecutive newlines, and only the first part is retained.
We then remove summaries whose word count is less than 10, greater than 512, or exceeds the length of the document.
Following this, the generated summary is as- sessed using the ROUGE metric to gauge its over- lap with the original document. In this case, we treat the original document the ground truth and the produced summary the prediction. As high-quality summary expected to encompass a subset of the document's 's information, the recall score largely depends on the document's length and is not particularly informative for evaluating summary quality. Consequently, we rely solely on precision scores for filterin. Specifically, we re- tain the summary only if its ROUGE-1 precision with the document exceeds 0.6 and its ROUGE-2 precision falls within the range of 0.25 to 0.8. We establish an upper bound for ROUGE-2 precision because we observed that the GPT model occasion- ally copies sentences from the original document, which may include excessive, irrelevant details.
Model Training
We proceed to train a seq2seq transformer language model ZCode++ (He et al., 2022) parameterized by A utilizing the GPTSUMM data. The objective is to minimize negative log-likelihood given the inputs with prompts [P:
The filtering process is crucial for ensuring that the generated summary is of high quality and perti- nent to the input document. Filtering is particularly important during the bootstrapping generation of data type (, where we filtered out 17% of the pro- duced summarics. The filtering rate significantly lower for other data types. For example, less than 5% of the summaries were filtered for data type 2, where one-shot in-context demonstration was applied.
L(0)- - Σ log P. vi-1) vi-11) (1)
where Pg is the probability distribution given by model with parameters 6. We train the model in a multi-task manner, where we mix all the data from
1 to 6 and sample from the data pool in every minibatch. To better balance the model towards general applications, we adjust the sampling ratio between each task to up-sample smaller data modes and down- larger ones. Analogous to the data generation in GPTSUMM, also truncate the in-context examples in data 55) and 6 as described in Section 3.4.
3.4 Succinet Demonstrations
Prior work (Brown et al., 2020a) has discovered that an increasing number of in-context examples can enhance the overall quality of generation. How-
In-Context Demonstrations in Distillation. To improve the GPT generation quality, most of the

---

data in GPTSUMM are generated in a fewshot man- ner. A natural way to train the downstream model is to keep the input same as GPT. We call the corresponding model SHHERTSUUM-CCOssiee However, the resulting model might not be good at zeroshot summarization since most data are in fewshot format. To obtain a model good at both zeroshot and fewshot summarization, we propose randomly convert some of the fewshot examples zeroshot examples. More specifically, we train another model where we randomly include to 4 in in-context examples by excluding examples in the prompts. We use zeroshot learning for data ! and D. 0 or I examples for 2 and 33, and O to examples for 5) We call the corresponding model INHERISUUMM-Balneed Note that data 6 does not involve GPT generation, and we always include in-context examples in our training.
hyperparameters in Sec. 5.1, and then introduce the main experiment results in Sec. 5.2.
5.1 Implementation Details
Hlyer-parameter
Value
Warmup Steps
10,000
Learning Rates
2e-5
Batch Size
144 (base), 120 (large)
Local Attention Window
256
Global Layers
6/11 (base). 11/23 (large)
Weight Decay
0.01
Training Steps
300k
Learning Rate Decay
Linear
Adam
le-6
Adam 31
0.9
Adam $2
0.999
Gradient Clipping
1.0
Beam search size
5
Table 3: Syper-parameters for Training INHERIT- SUMM.
4.1 Adapting to Application Scenarios
After training the INHERITSUMM model on GPT- SUMM, we adapt the INHERITSUMM to three dif- ferent summarization settings.
We use the ZCode++ model (He et al., 2022) the language model backbone for training INHERIT- SUMM. We choose ZCode++ over other pretrained models since it achieves state-of-art performance on summarization tasks after finetuning. We exper- imented with both the pretrained Z-Code++ base and Z. Z-Code++ large model. We train the model on the GPTSUMM corpus with seq2seq loss (1) for 300K steps, with maximum input length of 3072. We summarize the hyperparameters in Table 3. To reduce the memory usage we follow ZCode++ use Fusion-in-Encoder. We use two layers as the global attention layer and local layers have a win- dow size of 256.
Zeroshot and fewshot learning with prompts. The ability to adapt to different tasks with no or few examples is one of the most important char- acteristies of large models like GPT. We test the pretrained NHERITSUUM model in exactly the same way as GPT. For zeroshot learning, we ran- domly sample one instruction from PromptSource (Bach et al., 2022). For fewshot learning, we in- clude up to 4 examples (decided by the input doc- ument length) with succinet demonstrations from the corresponding training set.
Fewshot learning with prefix tuning, We follow the setting in Chen and Shuai (2021) and Chen et al. (2022) to prefix-tume our model. For every task t, we add K task-specific prefix vectors at every layer of the encoder and decoder for INHER- ITSUMM, parameterized by 0 Then we freeze all other parameters in INHERITŠUMM and tune 0f by minimizing (1).
Test settings. We test INHERIT' UMMM in 4 different settings: i) zeroshot learning with random instruc- tions from PromptSource (Bach et al.. 2022), ii) 4-shot learning with instructions, iii) 10-shot learn- ing with prefix tuning, and iv) fully supervised learning. i) and ii) are prompt- based settings typi- cally employed by large models like GPT, whereas iii) and iv) are more traditional settings for smaller models.
Fully Supervised Learning. In this setting we use all the data in downstream datasets to perform fully supervised finetuning. All the model parameters are finetuned by minimizing loss (1), without any prompts or instructions.
We compare with GPT the original ZCode++, BART (Lewis et al., 2019) and UniSumm (Chen et al., 2022). Due to hardware and compute limits, do not compare with GPT in prefix tuning and fully supervised settings. For fewshot learning with prefix tuning, We follow Li and Liang (2021) to tune prefix embeddings at every encoder and de-
Baselines.
Lext-ddainci-O00
5 Experiments
We detail the experiment results in this section. We first specify the implementation details and

---

Datasets
Domain
Avg. D/S Length
We choose ROUGE-2 as it has better correlation with human judgments on meta-cval benchmarks (Zhong et al., 2022). All the experiment results are from our own runs.
MultiNews
News
1,979 275
XWiki
Wikipedia
971 / 88
SAMSum
Dialogue
136 / 24
Zeroshot learning. INHERITSUMM models demonstrated better performance than all the base- lines in 3 out of 5 test datasets, well the av- erage over the datasets. INHERITSUMM'S perfor- mance is inferior to BART or UniSumm on Multi- News and Xwiki respectively, possibly because the summary on these two datasets are longer than the other datasets (this is also true for GPT-3.5 model), INHERIT gets higher performance than the teacher GPT-3.5 model. This is probably because INHERITSUMM is specialized in summarization, while GPT-3.5 might fail follow the instructions to summarize the input document.
Reddit-TIFU
Forum
496 29
BigPatent
Legal
2.853 119
Table 4: Statisties of testing datasets. Avg. D/S Length the average number GPT tokens for docu- ment/summary for the corresponding test set.
coder layer. For local layers in Fusion-in-Encoder of ZCode++ and INHERITSUMM, one set of prefix embeddings are inserted at every local window for every local layer.
Testing Datasets. We test on 5 summarization datasets on diverse domains (a summary in Table 4):
Among the four variants of INHERITSUMM, the base
the highest ROUGE score. The models trained the balanced way receive more zeroshot examples in its train- ing process, which probably makes them better at zeroshot lcarning. However, this is not true for INHERITSUMM-LLEn models, where the balanced model slightly behind the consistent model. This might be because large models are more capable when generalizing across different settings, and the data composition (whether balanced or consistent) is not very important for large models.
MultiNews (Fabbri et al., 2019) is a multi- document news summarization dataset with news from different sources.
XWiki (Perez-Beltrachiní and Lapata, 2021) is cross-lingual summarization dataset focusing Wikipedia articles. We use the English data with paired documents and summaries.
SAMSum (Gliwa et al., 2019) a dialogue summarization with chit-chat dialogues in online chatting styles like Messenger and WhatsApp. Both the dialogue and summary are human-written by expert linguists.
Fewshot learning with instructions. GPT-3.5 achieves the best average ROUGE score of 12.27 in this setting, whereas our INHERITSUMM models are only behind GPT-3.5 by a small gap. Among the four variants, INHERITS SUMM BLMMM-Balanced-Larr achieves the best score 12.15, slightly behind GPT-3.5.
Reddit-TIFU (Kim et al., 2019) is another dialogue summarization dataset focusing on the online forum Reddit. The language style on Reddit is significantly different from news articles.
BigPatent (Sharma et al., 2019) a legal document summarization dataset with US patent documents and numan-writted abstracts. Docu- ments come from 9 different domains.
also beats GPT-3.5 on 2 of the test datasets. Large mod- els are generally better in fewshot learning than base models. The performance between models trained with balanced or consistent data is com- parable, likely because both models receive large quantities of fewshot data in their training.
Some of the datasets have an extensive set that takes a long time test GPT on. Also, the GPT API has a content filter that rejects some of the input documents. Therefore, we randomly sam- ple 500 documents from the test sets, and choose those that pass GPT's content filter to compare the baselines.
Fewshot learning with prefix tuning. INHERIT- SUMM generally achieves the best performance in this setting, only loses to UniSumm on Bigpatent by a small margin. INHERIT UMMII WWLMMMMM-Csssstm base is the best in the average performance for the prefix tuning setting. The prefix tuning results of INHERITSUMM are also significantly better than the original ZCode++ models, suggesting the effec- tiveness of our distillation training.
5.2 Experiment Results
We summarize our main results for the four settings in Table 5. For simplicity and space reasons, we include ROUGE-2 scores in our experiment results.

---

Task
Datasets
7-Base
7-Large
GPT-3.5
BART
UniSumm
IS-Base-B
IS IS-Base-C
IS-Large--B
IS-Large-CC
Samsum
0.78
0.01
10.85
8.05
4.58
17.82
16.13
15.05
16.73
Xwiki
1,41
0.12
6.97
5.47
9.31
8.35
8.02
8,50
8.59
Zeroshot
Reddit Tifu
0.09
0.00
4.02
3.03
4.41
5.75
4.94
5.97
5.93
Bigparent
1.188
0.01
10.09
9.08
10.83
12.36
12.29
12.44
11.93
MultiNews
1,54
0.07
8.23
10.28
3.47
8.65
8.31
7.01
7.72
1.00
0.04
8.03
7.18
6.52
10.59
9.94
9.80
10.18
(insttot(nssrrcc)
Samsum
0.00
0.05
18.67
1.08
0.60
18.90
18.93
18.71
17.59
Xwiki
0.98
0.10
11.83
1.53
3.78
9.55
9.84
9.58
9.99
Reddit Tifu
0.03
0.03
6.68
0.62
0.91
7.28
7.14
8.51
7.93
Bigparent
1.22
0.01
12.85
3.55
4.45
9.98
12.71
13.18
12.88
MulliNews
0,92
0.07
11.32
1.92
1.33
10.21
9.76
10.77
10.04
Avg
0.63
0.05
12.27
1.74
2.21
11.19
11.67
12.15
11.68
Samsum
14.25
15.79
N/A
9.88
11.37
20.97
20.99
19.49
19.89
Fewshot(Prefix
Xwiki
10.47
12.07
N/A
11.08
8.29
11.91
12.07
12.20
12.46
Reddit Tifu
3,92
3.94
N/A
2.78
6.92
5.26
Bigpatent
7.13
5.86
N/A
6.99
13.12
12.23
12.65
12.84
12.23
MulliNews
4.88
8.92
N/A
11.63
10.84
11.69
12.06
10.91
11.33
Avg
8.13
9.32
N/A
8.47
9.96
12.63
12.86
12.47
12.23
Samsum
27.49
27.91
N/A
29.26
22.36
30.12
29.87
28.52
28.60
Supervised
Xwiki
21.93
21.77
N/A
20.21
18.05
21.70
21.74
22.48
20.63
Reddit
10.66
10.37
N/A
11.33
8.42
11.57
10.25
10.31
Bigpatent
17.94
12.64
N/A
17.88
17.38
17.99
18.03
21.67
23.11
MultiNews
17.66
19.11
N/A
17.87
18.69
19.83
20.38
19.03
201,5
Avg 19.14
18.36
N/A
19.31
16.98
20.24
20.35
20.39
20.65
Samsum
10.63
10.94
N/A
12.07
9.73
21.95
21.48
20.44
20.70
Xwiki
8.70
8.52
N/A
9.57
9.86
12.88
12.92
13.19
12.92
Mean
Reddit Tifu
3.67
3.58
N/A
4.44
4.98
7.74
7.59
7.91
7.36
Bigpatent
6.87
4.63
N/A
9.38
11.45
13.14
13.92
15.03
15.04
MultiNews
6.25
7.04
N/A
10.43
8.58
12.59
12.63
11.93
12.42
Avg 7.23
N/A
9.18
8.92
13.70
13.69
Table 5: Main Experiment Results. For simplicity, we only include ROUGE-2 scores in the table. stands for INHERITSUMA B stands for balanced and C stands for consistent. The largest and second-largest number in each row are in bold and underlined respectively. The "Mean" section at the bottom of the table is the mean performance over 4 different settings.
Fully supervised learning. Lastly, INHERIT- SUMM outperforms all the bselines in the fully su- pervised learning setting well. INHERITSUMM outperforms the original ZCode++ model, show- ing the transfer ability of our distillation training. Among four variant INHERITSUMM, INHER- SSSUMM--Consstte--Laage gives the best perfor- mance. This is likely because large models are more powerful with fully supervised data, and con- sistent data training is better for the transfer of knowledge.
Datasets R- R-2(zeroshot)
R-2 (fewshot)
9.82
10.69
10.02
10.04
10.12
3.06
8,80
9,95
6.53
6.17
All
10.59
11.19
Table Performance of single-dataset training. R-2 stands for ROUGB-2 scores. All scores are averaging over test sets.
5.2.1 Analysis
For average over the 4 settings, INHERITSUMM strongly outperforms all the baselines the aggre- gate score over 4 settings, showing that INHERIT- SUMM is the most versatile model across different training scenarios. The average performance of the 4 variants is quite close.
Effect of different training datasets. One natural question is about the effect of each part of GPT- SUMM in SUMM''S performance. While is not possible to test every combination ! 6, we follow the FLAN paper (Wei et al., 2021)

---

and ZCode++ (He al.. 2022) focuses on super- vised summarization. PEGASUS (Zhang al., 2020) proposes pretraining methods for unsuper- vised summarization. UniSumm (Chen et al., 2022) explores fewshot summarization with BART-Large (Goyal ct al., 2022) explores zeroshot and fewshot summarization for GPT-3. To the best of our knowl- edge, we are the first paper to explore the gener- alization over zeroshot, fewshot, and supervised summarization.
Effect InheritSumm
12.0
Bass Large
11.5
souera
10.0
9.5
Model Distillation from GPT There has been several works distilling knowledge from the GPT series of models. Wang et al. (2022) finetunes LLaMA (Touvron et al., 2023) with 52K cnstruction-follvviii text dext-davinci-- -003 variant of GPT-3.5. It shows that the resulting Alpaca model behaves sim- ilarly to GPT-3.5 on instruction-following evalua- tion suite. Peng et al. (2023) further improves the performance by using instructions from the GPT-4 model. However, all these works focuses on the general setting with general user instructions. To the best of our knowledge, we are the first work on distillation from GPT-3/4 that focuses par- ticular task. Our focus summarization makes us able to smaller models while not losing too much performance.
9.0
0.0
0.5
1.0
1.5
2.0
3.0
Shot Number
Figure 2: Effect of number of shots for INHERITSUMM base and large model.
data using the
to test the performance of INHERIT under individual parts. In Table 6, we train a base INHER- ITSUMM in the balanced setting with the same hy- perparameters on I + 2 3 ④ 5 respec- tively. The results show that all the GPT-generated data ① (5) gives better zeroshot/fewshot per- formance than supervised datasets (6), except for 4 on fewshot: this is expected because 4 contains only zeroshot training data. As all parts of data can help boost the zeroshot/fewshot per- formance, we include all of them as our training data. Effect of succinet demonstrations. In order to test the effect of succinct demonstrations, we test INHERIT S SUMM-Balancedls performance with different number of shots. In Figure we plot the performance of base and large INHERITSUMM- Balancedfrom 0-shot to 4-shots For both models, the performance improves from 1-shot to 4-shots. For the large model, the performance also improve when we go from zero-shot to 1-shot, but this is not the case for base model. This shows that us- ing the traditional one-shot method may even hurt the performance, possibly due model capacity reasons. Our succinet prompt method can always incorporate more in-context examples and improve the model's performance.
7 Conclusion & Future Works
We propose INHERITSUMM by distilling knowl- edge from GPT-3.5 using its summary on a broad range of documents. Base model INHERIT- SUMM with only 400M parameters exhibits ver- satile capability on zeroshot, fewshot, and fully su- pervised summarization. surpassing performance of previous small models and beats or rivals GPT- 3.5 in the overall performance.
For future works, it would be interesting to inves- tigate the performance when distilling from other variants of GPT, like TEXT-DAVINCI-ON or GPT- 4. Another interesting direction is controllable summarization by using proper instructions, IN- HERITSUMM can be further trained to generate customized summarizations with style or length constraints.
6 Related Works
Text Summarization has been extensively ex- plored by the community (Nenkova and McKeown, 2012). Previous works mostly focus one or settings of renoshoffeewhhotsspprviis learn- ing only. For example, BART (Lewis et al., 2019)
Limitations
One limitation of INHERITSUMM that it is only distilled in English and therefore lacks the multilin- gual generalization capability in other languages. Another limitation is that the distillation process
We combine ① and 2 hecause they are quite similar in style and focus on the same set of input documents.

---

of INHERITSUMM is cost inefficient as it requires extensive API calls GPT.
American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, Volume 2 (Short Pupers), pages 615 615-621, 121,, New Or- leans, Louisiana. Association for Computational Lin- guistics.
References
Stephen H. Bach, Victor Sanh, Zheng-Xin Yong, Albert Webson, Colin Raffel, Nihal V. Nayak, Abheesht Sharma, Taewoon Kim, M Saiful Bari, Thibault Fevry, 7aid Alyafeai, Manan Dey, Andrea San- tilli, Zhiqing Sun, Srulik Ben-David, Canwen Xu, Gunjan Chhablani, Han Wang. Jason Alan Fries, Maged S. Al-shaibani, Shanya Sharma, Urmish Thakker, Khalid Almubarak, Xiangru Tang, Xian- gru Tang, Mike Tian-Jian Jiang, and Alexander M. Rush, 2022. Promptsource: An integrated develop- ment environment and repository for natural language prompts.
Alexander Fabbri, Irene Li, Tianwei She, Suyi Li, and Dragomir Radev. 2019. Multi-news: A large-scale multi docum-d summarization dataset and abstrac- tive hierarchical model. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1074-1084, Florence, Italy, Asso- ciation for Computational Linguistics.
Leo Gao, Stella Biderman, Sid Black, Laurence Gold- ing, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. 2020. The Pile: An 800gb dataset diverse text for language modeling. arXiv preprint arXiv:2 101.0000..
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss. Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler. Jeffrey Wu. Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020a. Language models are few-shot learners. In Ad- vances in Neural Information Processing Systems, volume 33, pages 1877-1901 Curran Associates, Inc.
Bogdan Gliwa, Iwona Mochol, Maciej Biesek, and Alek- sander Wawer. 2019. SAMSum corpus: A human- annotated dialogue datasct for abstractive summa- rization. In Proceedings of the 2nd Workshop New Frontiers Summarization, pages 70-79, Hong Kong, China. Association for Computational Linguis- tics.
Tanya Goyal, Junyi Jessy Li, and Greg Durrett. 2022. News summarization and evaluation in the era of gpt-3, arXiv preprint arXiv:2209. 12356.
Pengcheng He, Baolin Peng. Liyang Lu, Song Wang, Jie Mei, Yang Liu, Ruochen Xu, Hany Hassan Awadalla, Yu Shi, Chenguang Zhu, et al. 2022. 7-code++: A pre- pre-trained language model optimized for abstractive summarization. arXiv preprint arXiv:2208.091)
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020b. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901.
Byeongchang Kim, Hyunwoo Kim, and Gunhee Kim. 2019. Abstractive summarization of Reddit posts with multi-level memory networks. In Proceedings of the 2019 Conference of the North American Chap- of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pages 2519-2531. Minneapolis, Min- nesota. Association for Computational Linguistics.
Yi-Syuan Chen and Hong-Han Shuai. 2021. Meta- transfer learning for low-resource abstractive sum- marization. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 12692- 12700.
Yulong Chen, Yang Liu, Ruochen Xu, Ziyi Yang. Chen- guang Zhu, Michael Zeng, and Yue Zhang. 2022. Unisumm: Unified few-shot summarization with multi-task pre-training and prefix-tuning. arXiv preprint 2rXiv:2211.0977..
Mahnaz Koupace and William Yang Wang. 2018. Wiki- how: A large scale text summarization dataset. arXiv preprint 1810.09305
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, AbdeIrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019. Bart: De- noising sequence-Ko-sequenee pre-training for natural language generation, translation, and comprehension. arXiv preprint 1rXiv::110.13446,
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2022. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311.
Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arxiv:2101.00/90.
Arman Cohan, Franck Dernoncourt, Doo Soon Kim, Trung Bui, Seokhwan Kim, Walter Chang, and Nazli Goharian. 2018. A discourse-aware attention model for abstractive summarization of long documents. In Proceedings of the 2018 Conference of the North
Percy Liang. Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian

---

Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Ku- mar, et al. 2022. Holistic evaluation of language models. arXiv preprint arXiv:2211.088
Ming Zhong, Yang Liu, Da Yin, Yuning Mao, Yizhu Jiao, Pengfei Liu, Chenguang Zhu, Heng Ji, and Jiawei Han. 2022. Towards unified multi- dimensional evaluator for text generation. In Pro- ceedings of the 2022 Conference on Empirical Meth- ods in Natural Language Processing, pages 2023- 2038, Abu Dhabi, United Arab Timirates. Association for Computational Linguistics.
Ani Nenkova and Kathleen McKeown, 2012. A survey of text summarization techniques. Mining text data, pages 43-76.
Baolin Peng, Chunyuan Li, Pengcheng He, Michel Gal- ley. and Jianfeng Gao. 2023. Instruction tuning with gpt-4, arXiv preprint urXiv:2304.03277
Chenguang Zhu and Michael Zeng. 2022. Impossible triangle: What's 's next for pre-trained language mod- els? arXiv preprint arXi::2204.06.30.
Laura Berez-Beltrachini and Mirella Lapata. 2021. Mod- els and datasets for cross-lingual summarisation. In Proceedings of the 2021 Conference on Empirical Methods Natural Language Processing, pages 9408-9423, Online and Punta Cana, Dominican Re- public. Association for Computational Linguisties.
Abigail See, Peter J. Liu, and Christopher D. Manning. 2017. Get the point: Summarization with pointer- generator networks. nual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1073- 1083, Vancouver, Canada. Association for Computa- tional Linguistics.
Proceedings of 55th
Eva Sharma, Chen Li, and Lu Wang. 2019. BIG- PATENT: A large-scale dataset for abstractive and coherent summarization.
Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2204-2213, Florence, Italy. Asso- ciation for Computational Linguistics.
Shaden Smith, Mostofa Patwary, Brandon Norick, Patrick LeGresley, Samyam Rajbhandari, Jared Casper, Zhun Liu, Shrimai Prabhumoye, George Zerveas, Vijay Korthikanti, al. 2022. Using deep- speed and megatron to train megatron-turing nlg 530b, large-scale generative language model. arXiv preprint arXiv:2201.I 1990.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux. Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, al. 2023. Llama: Open and effi- cient foundation language models. arXiv preprint arXiv:2302.1323 139.71
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Al- isa Liu, Noah A Smith. Daniel Khashabi, and Han- nanch Hajishirzi. 2022. Self-instruct: Aligning lan- guage model with self generated instructions. arXiv preprint 2212: 10560.
Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Du, drew M Dai, and Quoe V Le. 2021. Finetuned lan- guage models are zero-shot learners. arXiv preprint arXiv:2109.0165..
Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Pe- ter Liu. 2020. Pegasus: Pre-training with extracted gap-sentences for abstractive summarization. In In- lernational Conference on Machine Learning, pages 11328-11339, PMLR.

---

A
6 individual tasks. They are mixed by the ratio of List of Prompts
Below we list the prompts that we use from PromptSource. [doc] stands for the input document. [doc] Write a summary of the text above Summary:
[doc] How would you rephrase that in a few words? Rephrase:
My college roomnate asked me what this article means: [doc] So I recapped it in layman's s terms:
Summarize this document: [doc] Summary:
[doc] Given the above document, write sentence to summarize:
Summary:
First, please read the article below. [doc] Now, can you write me an extremely short abstract for it? An extremely short abstract:
[doc] TL;DR:
Can you write an outline of the following article in few points?
Article: [doc] Outline:
Summarise the article: [doc] Summary:
In 2 or 3 sentences, what are the main points should remember
from this news article? Article: [doc] Main points:
Could you please generate TLDR (Too Long Didn't Read) summary of the following news article? Article: [doc] TLDR summary:
Condense the article down to the essentials to present it the form of short cards in mobile news apps: [doc] Essentials:
Sum the following article in brief: [doc] Breifs:
Extract key points from the article based on which the stock market could react: [doc] Key points:
Summarize this document: [doc] Summary:
[doc] Given the above document, write summary. Summary:
Summarize: [doc] Summary:
[doc] To sum up this document:
Sum up the following document: [doc] Summary:
what are the key points across these news articles: Article: [doc] Key points:
Synthesize these documents into single one: [doc] Summary: I want to edit the following articles into more concise summary:
Article: [doc] Summary:
Write a summary of the following articles: Document: [doc] Summary:
I'm m trying to distill these articles down into one: Article: [doc]
Summary:
The special "following" prompt is
"Follow the example (s) above and summarize the document below:
Document: [doc] Summary:
The in-context examples are prepended to the prompt in the format of Document: [doc]
Summary: (sum] where [doc] and [sum] are the document and summary for the in-context examples respectively.
B Sampling ratio of tasks
We mix ① and 2 as one task, and treat 4 1,0.1..110.15,..11... 0.15].
Sample Outputs
We list outputs from INHERITSUMM and baselines, together with the groundtruth human summaries. We can see that the outputs from INHERITSUMM follow GPT-3.5 and are often equal or better quality.

---

GPT-3.5 sometimes fails to follow the instructions, and INHERITSUMM avoids such problems.
System
Summary
The Rod of Asclepius is symbol associated with the Greek god Asclepius, who was associated with healing and medicinal arts in Greek mythology. The symbol consists of a snake and staff, which were sometimes depicted separately in antiquity. The most famous temple of Asclepius was at Epidaurus in north-eastern Peloponnese. The Rod of Asclepius has been used logo by number of organizations and services.
IS- IS-Large- C Zeroshot
The Rod of Asclepius is symbol associated with the Greek god Asclepius, deity associated with healing and medicinal arts in Greek mythology. The symbol consists of snake wrapped around a staff, and is thought to represent the dual nature the work the physician, who deals with life and death, sickness and health. The Rod of Asclepius is used as logo by a number of organizations and services.
LS-Large-C Fewshot
GPT-3.5 Zeroshot
the Rod of Asclepius.
The Rod of Asclepius is staff with snake wrapped around it that used as symbol of healing. It is named after the Greek god Asclepius, who was associated with healing and medicinal arts in Greek mythology. The symbol has been used by a number of organizations and services that are related to healing.
GPT-3.5 Fewshot
In Greek mythology. the Rod of Asclepius (.. sometimes also spelled Asklepios or Asclepius), also known as the Staff of Aesculapius and as the asklepian, is a serpent-entwined rod wielded by the Greek god Asclepius, deity associated with healing and medicine. The symbol has continued to be used in modern times, where associated with medicine and health care, yet frequently confused with the staff of the god Hermes, the caduceus. Theories have been proposed about the Greek origin of the symbol and its implications.
Human
Charlotte Louise Juliette de Monaco was the illegitimate daughter of Marie Juliette Louvet, a cabaret singer, and Louis, Hereditary Prince of Monaco and Duke of Valentinois. She was adopted by Louis in 1919 and became heir presumptive to the throne as Hereditary Princess when her grandfather died and her father inherited the princely crown in 1922. Charlotte married Count Pierre de Polignac in 1920, but the marriage was happy one and they divorced in 1933. In 1944, Charlotte renounced and ceded her rights to the throne her son Rainier. She died in 1977 in Paris.
IS-Large-C Zeroshot

---

Table 7 continued from previous page
System
Summary
Charlotte Louise Juliette de Monaco was the illegitimate daughter of Marie Juliette Louvet, a cabaret singer, and Louis, Hereditary Prince of Monaco and Duke of Valentinois, and heir of Monaco's ' reigning monarch, Prince Albert I. In 1911, law was passed recognising Charlotte as Louis's daughter, and declaring her to be a dynastic member of the sovereign family. In 1919, Louis adopted Charlotte in Paris, and she became heir presumptive to the throne as Hereditary Princess when her grandfather died and her father inherited the princely crown in 1922. Charlotte married Count Pierre de Polignac in 1920, but the marriage was not happy one and the couple divorced in 1933. In 1944, Charlotte renounced and ceded her rights the throne to her Rainier, subject to the stipulation that he did predecease her. Charlotte died in 1977 in Paris.
IS- IS-Large-C Fewshot
Charlotte was illegitimate child of the heir to the throne of Monaco. In order legitimize her and prevent succession crisis, she was married off to a Frenchman who took the Grimaldi name. The couple had two children before divorcing. Charlotte later moved to Paris, where she died.
GPT-3.5 Zeroshot
Princess Charlotte Monaco was the illegitimate daughter Marie Juliette Louvet, cabaret singer, and Louis, Hereditary Prince of Monaco and Duke of Valentinois, son and heir of Monaco's 's reigning monarch, Prince Albert I. Louis had no legitimate children or siblings, so Charlotte was adopted into the dynasty in order to prevent succession crisis. She married Count Pierre de Polignac of Hennebont, Morbihan, Brittany, France, who took the surname Grimaldi and became a Prince of the Monegasque Royal Family, The couple had two children, but their marriage was not a happy one and they divorced in 1933. Princess Charlotte renounced her rights to the throne in 1944, and she died in 1977.
GPT-3.5 Fewshot
Princess Charlotte Monaco, Duchess of Valentinois (Charlotte Louise Juliette Grimaldi; 30 September 1898 16 November 1977), was the daughter of Louis 11, Prince of Monaco, and the mother of Rainier III, Prince of Monaco. From 1922 until 1944, she was the Hereditary Princess of Monaco, heir presumptive to the throne.
Human
The article is discussing pedal assembly that is used to control the throttle of a vehicle. The assembly includes mounting bracket, guide structure, carrier assembly, a drive assembly, a pedal assembly, a resistance assembly, and generator. The pedal assembly is adjustable accommodate operators varying anatomical dimensions, and the resistance assembly provides damping for a -djjuaaut pedal assembly.
IS-Large-C Zeroshot

---

Table 7 continued from previous page
System
Summary
The invention accelerator pedal assembly 10 includes mounting bracket 1, guide structure 12, carrier assembly 13, drive assembly 14, pedal assembly 16, resistance assembly 18, and generator means 20. The pedal assembly 10 is intended to allow efficient fore and aft movement of the pedal assembly accommodate operators of varying anatomical dimension and operative to generate an electronic or drive by wire signal in response to pivotal movement of the pedal assembly while retaining the same ergometric operation of the pedal irrespective of the position of adjustment of the pedal.
IS-Large-C Fewshot
The invention pedal assembly provides the desired adjustability, the desired ergometrics, and the desired hysteresis effect in structure that is simple, inexpensive, and positive and reliable in operation.
GPT-3.5 Zeroshot Zerosht
The invention provides a pedal assembly for motor vehicle that can be adjusted to accommodate operators of varying anatomical dimensions, that has constant ergometries irrespective of the position of adjustment of the pedal structure, and that provides desired hysteresis effect in any position adjustment the pedal structure. The pedal assembly is simple, inexpensive, and positive and reliable in operation.
GPT-3.5 Fewshot
an electronic adjustable control pedal assembly for a motor vehicle including a carrier, a guide rod adapted to be secured to the dash panel of the vehicle and mounting the carrier for fore and aft movement along the guide rod, a power drive operative to move the carrier along the guide rod. pedal arm pivotally mounted on the carrier, potentiometer mounted the carrier and operative to generate an output electrical signal proportioned the extent of pivotal movement the pedal arm, and resistance assembly to provide feedback or feel to the operator - the resistance assembly includes an annular friction surface defined on the carrier, sleeve positioned over the friction surface, and a coil spring encircling the sleeve and arranged to be tightened in response to pivotal movement of the pedal arm to squeeze the sleeve against the friction surface and generate friction resistance force the friction resistance force adds the torsional resistance force the spring during application the pedal and subtracts from the torsional resistance force of the spring upon release of the pedal, whereby to create hysteresis effect.
Human
Table Example zeroshot fewshot outputs from IS-Large-C and GPT-3.5, together with ground truth human summaries.


Reviews:
[{"review_id": "4MyHXvKgKL", "reviewer": "Reviewer_FFF7", "summary": "", "questions": "Question A:\nWhy did you prefer ROUGE-2 instead ROUGE-L?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the insightful comments, and we are glad that you also agree on GPTSumm’s great value in developing universal summarization models. Please see our response to your concerns below.\n\n1. **Not enough contribution for the InheritSumm model**: We respectfully disagree with this argument. While the GPTSumm dataset is indeed a significant contribution, InheritSumm serves as a versatile, compact model capable of summarizing a wide range of document styles. The model's release is pivotal for the research community and opens avenues for future work. Method-wise, we also propose to use succinct demonstrations for distilling from GPT-3 on the summarization task (Sec 3.4). Given that summarization tasks often involve long inputs, this distillation method is particularly crucial. To the best of our knowledge, the method is novel for summarization distillation.\n\n2. **Comparison between base and large models**: We acknowledge the observation made in Table 5 regarding the performance similarities between the base and large models. Our hypothesis is that this may be attributed to the inherent characteristics of the ZCode++ models. In our experiments, the base model has shown competitive performance across various tasks, not just in summarization. Nevertheless, the fact that the base model performs on par with the large while outperforming other models is a benign phenomenon - It suggests that excellent performance can be achieved with a more resource-efficient model.\n\n3. **Question A: ROUGE-2 vs ROUGE-L**: We choose ROUGE-2 over other metrics because previous studies [1] find that ROUGE-2 has a better correlation with human judgments on meta-eval benchmarks like SummEval, and SummEval has a similar composition of datasets as our testing benchmark. We will make this point clear in the final version.\n\nWe hope this addresses your concerns and questions adequately. Should you have any further queries, please do not hesitate to ask.\nThank you for your time and consideration.\n\n[1] Zhong, Ming, et al. \"Towards a unified multi-dimensional evaluator for text generation.\" arXiv preprint arXiv:2210.07197 (2022).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The new GPTSUMM dataset will be useful for those in community working on summarization. The model trained using this dataset produces summarization output comparable to GPT-3.5.", "reasons_to_reject": "For me, this paper contains enough work to be accepted as a short paper but not enough for a long paper. The paper is titled InheritSumm which is misleading. The main contribution of this paper is the GPTSUMM dataset. For training the InheritSumm models the authors simply applied exiting  ZCode++ on the GPTSUMM dataset. \n\nI would have still considered it about InheritSumm if the author included sufficient analysis of the InheritSumm results. For example, as shown in Table 5, there is basically no improvement between InheritSumm-base and InheritSumm-large models. In fact,  InheritSumm-base performs slightly better than InheritSumm-base-large. The authors mentioned they use ZCode++ for training both models using their dataset. But they did explain the peculiarity in the results. ", "questions_to_authors": "Question A:\nWhy did you prefer ROUGE-2 instead ROUGE-L?", "paper_topic_and_main_contributions": "Although the paper is titled InheritSumm, it is actually about a new dataset GPTSUMM that the authors created. They showed that if an existing model such as  ZCode++ is trained on this dataset, the resulting model's output for the summarization task is comparable to that of GPT-3.5.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Page 1\n\nin the the way => in the way"}, "extra_text": {}}, {"review_id": "HzjAWUycz1", "reviewer": "Reviewer_mfxh", "summary": "", "questions": "1. Could you discuss more the effect of the choice of demonstration examples on model performance? \n2. Could you discuss some of the qualitative differences in your model output across tasks and datasets?\n3. The reader can estimate it, but if allowed, can you state the cost for the API calls to create the dataset?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the insightful comments. Please see our response to your questions below.\n\n1）**Effect of the choice of demonstration examples**: We deliberately make the model robust to different types of demonstration examples by varying the demonstration example types when collecting the GPTSumm dataset (see Table 2). As a result, the InheritSumm model can take in a variety of demonstration examples not necessarily in the same style as the input document, and follow the demonstration to produce a new summary in the desired style. We also did an ablation study in Figure 2 on the number of demonstration examples. Generally, more demonstration examples help, and our succinct demonstration method is critical to generating high-quality summaries.\n\n2）**Qualitative analysis**: We will include additional case studies in our final version. Below are some example outputs. The outputs from InheritSumm follow the pattern of GPT-3.5 and are often of equal or better quality. GPT sometimes fails to follow the instructions, and InheritSumm avoids such problems.\n\n3）**Cost estimate**: We cannot reveal the exact cost due to confidentiality reasons, but we can give an estimate based on the public pricing. Our full dataset contains about 6.4M summaries generated by GPT3 (excluding existing datasets in part 6 in Table 2) In total there are 13 billion total input tokens including demonstration, and 1.6 billion output tokens for summary. This costs about $23k US dollars according to the current pricing of GPT-3 API.\n\n# Case Study\n\n\n| System | Summary |\n| --- | --- |\n| IS-Large-C Zeroshot | The Rod of Asclepius is a symbol associated with the Greek god Asclepius, who was associated with healing and medicinal arts in Greek mythology. The symbol consists of a snake and a staff, which were sometimes depicted separately in antiquity. The most famous temple of Asclepius was at Epidaurus in north-eastern Peloponnese. The Rod of Asclepius has been used as a logo by a number of organizations and services. |\n| IS-Large-C Fewshot | The Rod of Asclepius is a symbol associated with the Greek god Asclepius, a deity associated with healing and medicinal arts in Greek mythology. The symbol consists of a snake wrapped around a staff, and is thought to represent the dual nature of the work of the physician, who deals with life and death, sickness and health. The Rod of Asclepius is used as a logo by a number of organizations and services. |\n| GPT-3.5 Zeroshot | the Rod of Asclepius. |\n| GPT-3.5 Fewshot | The Rod of Asclepius is a staff with a snake wrapped around it that is used as a symbol of healing. It is named after the Greek god Asclepius, who was associated with healing and medicinal arts in Greek mythology. The symbol has been used by a number of organizations and services that are related to healing. |\n| Human | In Greek mythology, the Rod of Asclepius (,, sometimes also spelled Asklepios or Asclepius), also known as the Staff of Aesculapius and as the asklepian, is a serpent-entwined rod wielded by the Greek god Asclepius, a deity associated with healing and medicine. The symbol has continued to be used in modern times, where it is associated with medicine and health care, yet frequently confused with the staff of the god Hermes, the caduceus. Theories have been proposed about the Greek origin of the symbol and its implications. |\n\n| System | Summary |\n| --- | --- |\n| IS-Large-C Zeroshot | Charlotte Louise Juliette de Monaco was the illegitimate daughter of Marie Juliette Louvet, a cabaret singer, and Louis, Hereditary Prince of Monaco and Duke of Valentinois. She was adopted by Louis in 1919 and became heir presumptive to the throne as Hereditary Princess when her grandfather died and her father inherited the princely crown in 1922. Charlotte married Count Pierre de Polignac in 1920, but the marriage was not a happy one and they divorced in 1933. In 1944, Charlotte renounced and ceded her rights to the throne to her son Rainier. She died in 1977 in Paris. |\n| IS-Large-C Fewshot | Charlotte Louise Juliette de Monaco was the illegitimate daughter of Marie Juliette Louvet, a cabaret singer, and Louis, Hereditary Prince of Monaco and Duke of Valentinois, son and heir of Monaco's reigning monarch, Prince Albert I. In 1911, a law was passed recognising Charlotte as Louis's daughter, and declaring her to be a dynastic member of the sovereign family. In 1919, Louis adopted Charlotte in Paris, and she became heir presumptive to the throne as Hereditary Princess when her grandfather died and her father inherited the princely crown in 1922. Charlotte married Count Pierre de Polignac in 1920, but the marriage was not a happy one and the couple divorced in 1933. In 1944, Charlotte renounced and ceded her rights to the throne to her son Rainier, subject to the stipulation that he did not predecease her. Charlotte died in 1977 in Paris. |\n| GPT-3.5 Zeroshot | Charlotte was an illegitimate child of the heir to the throne of Monaco. In order to legitimize her and prevent a succession crisis, she was married off to a Frenchman who took the Grimaldi name. The couple had two children before divorcing. Charlotte later moved to Paris, where she died. |\n| GPT-3.5 Fewshot | Princess Charlotte of Monaco was the illegitimate daughter of Marie Juliette Louvet, a cabaret singer, and Louis, Hereditary Prince of Monaco and Duke of Valentinois, son and heir of Monaco's reigning monarch, Prince Albert I. Louis had no legitimate children or siblings, so Charlotte was adopted into the dynasty in order to prevent a succession crisis. She married Count Pierre de Polignac of Hennebont, Morbihan, Brittany, France, who took the surname Grimaldi and became a Prince of the Monegasque Royal Family. The couple had two children, but their marriage was not a happy one and they divorced in 1933. Princess Charlotte renounced her rights to the throne in 1944, and she died in 1977. |\n| Human | Princess Charlotte of Monaco, Duchess of Valentinois (Charlotte Louise Juliette Grimaldi; 30 September 1898 - 16 November 1977), was the daughter of Louis II, Prince of Monaco, and the mother of Rainier III, Prince of Monaco. From 1922 until 1944, she was the Hereditary Princess of Monaco, heir presumptive to the throne. |\n\n| System | Summary |\n| --- | --- |\n| IS-Large-C Zeroshot | The article is discussing a pedal assembly that is used to control the throttle of a vehicle. The assembly includes a mounting bracket, a guide structure, a carrier assembly, a drive assembly, a pedal assembly, a resistance assembly, and a generator. The pedal assembly is adjustable to accommodate operators of varying anatomical dimensions, and the resistance assembly provides damping for a non-adjustable pedal assembly. |\n| IS-Large-C Fewshot | The invention accelerator pedal assembly 10 includes a mounting bracket 11, a guide structure 12, a carrier assembly 13, a drive assembly 14, a pedal assembly 16, a resistance assembly 18, and a generator means 20. The pedal assembly 10 is intended to allow efficient fore and aft movement of the pedal assembly to accommodate operators of varying anatomical dimension and is operative to generate an electronic or drive - by - wire signal in response to pivotal movement of the pedal assembly while retaining the same ergometric operation of the pedal irrespective of the position of adjustment of the pedal. |\n| GPT-3.5 Zeroshot | The invention pedal assembly provides the desired adjustability, the desired ergometrics, and the desired hysteresis effect in a structure that is simple, inexpensive, and positive and reliable in operation. |\n| GPT-3.5 Fewshot | The invention provides a pedal assembly for a motor vehicle that can be adjusted to accommodate operators of varying anatomical dimensions, that has constant ergometrics irrespective of the position of adjustment of the pedal structure, and that provides a desired hysteresis effect in any position of adjustment of the pedal structure. The pedal assembly is simple, inexpensive, and positive and reliable in operation. |\n| Human | an electronic adjustable control pedal assembly for a motor vehicle including a carrier , a guide rod adapted to be secured to the dash panel of the vehicle and mounting the carrier for fore and aft movement along the guide rod , a power drive operative to move the carrier along the guide rod , a pedal arm pivotally mounted on the carrier , a potentiometer mounted on the carrier and operative to generate an output electrical signal proportioned to the extent of pivotal movement of the pedal arm , and a resistance assembly to provide feedback or feel to the operator . the resistance assembly includes an annular friction surface defined on the carrier , a sleeve positioned over the friction surface , and a coil spring encircling the sleeve and arranged to be tightened in response to pivotal movement of the pedal arm to squeeze the sleeve against the friction surface and generate a friction resistance force . the friction resistance force adds to the torsional resistance force of the spring during application of the pedal and subtracts from the torsional resistance force of the spring upon release of the pedal , whereby to create a hysteresis effect . |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Both the dataset and its methodology will be useful for future work and can inspire task-specific distillation studies. \n2. The performance compares strongly against previous small models and GPT3.5", "reasons_to_reject": "1. Analysis of model outputs would be helpful to better contextualize model performance. Only ROUGE is reported and discussed, without mention of other aspects such as factual consistency. ", "questions_to_authors": "1. Could you discuss more the effect of the choice of demonstration examples on model performance? \n2. Could you discuss some of the qualitative differences in your model output across tasks and datasets?\n3. The reader can estimate it, but if allowed, can you state the cost for the API calls to create the dataset?", "paper_topic_and_main_contributions": "\nThe paper proposes a model distilled from GPT3.5 (text-davinci-002) on their curated GPT-SUMM dataset. The dataset consists of documents from the Pile corpus and supervised dataset and summaries are generated in several settings, including with/without in-context examples. The paper examines shortening demonstration documents to allow for more examples in few-shot learning. A ZCode++ model is trained on this data and compared in both zero and few-shot summarization settings. ", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "Not missing since it's concurrent work, but in case you missed it: On Learning to Summarize with Large Language Models as References\n"}, "extra_text": {}}, {"review_id": "zyfGgYxRPj", "reviewer": "Reviewer_UWoN", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the insightful comments. Please see our response to your concerns below.\n\n1）**Validity of the results**: While we are not able to include extra crowdsourced evaluations due to time and budget constraints, we will include extra case studies in the final version. Below are some examples. We can see that the outputs from InheritSumm follow GPT-3.5 and are often of equal or better quality. GPT sometimes fails to follow the instructions, and InheritSumm avoids such problems.\n\n2）**Reproducibility**: We will share our data and model upon acceptance of the paper. \n\nPlease let us know if you have further questions! Thank you.\n\n# Case Study\n\n\n| System | Summary |\n| --- | --- |\n| IS-Large-C Zeroshot | The Rod of Asclepius is a symbol associated with the Greek god Asclepius, who was associated with healing and medicinal arts in Greek mythology. The symbol consists of a snake and a staff, which were sometimes depicted separately in antiquity. The most famous temple of Asclepius was at Epidaurus in north-eastern Peloponnese. The Rod of Asclepius has been used as a logo by a number of organizations and services. |\n| IS-Large-C Fewshot | The Rod of Asclepius is a symbol associated with the Greek god Asclepius, a deity associated with healing and medicinal arts in Greek mythology. The symbol consists of a snake wrapped around a staff, and is thought to represent the dual nature of the work of the physician, who deals with life and death, sickness and health. The Rod of Asclepius is used as a logo by a number of organizations and services. |\n| GPT-3.5 Zeroshot | the Rod of Asclepius. |\n| GPT-3.5 Fewshot | The Rod of Asclepius is a staff with a snake wrapped around it that is used as a symbol of healing. It is named after the Greek god Asclepius, who was associated with healing and medicinal arts in Greek mythology. The symbol has been used by a number of organizations and services that are related to healing. |\n| Human | In Greek mythology, the Rod of Asclepius (,, sometimes also spelled Asklepios or Asclepius), also known as the Staff of Aesculapius and as the asklepian, is a serpent-entwined rod wielded by the Greek god Asclepius, a deity associated with healing and medicine. The symbol has continued to be used in modern times, where it is associated with medicine and health care, yet frequently confused with the staff of the god Hermes, the caduceus. Theories have been proposed about the Greek origin of the symbol and its implications. |\n\n\n\n\n\n| System | Summary |\n| --- | --- |\n| IS-Large-C Zeroshot | Charlotte Louise Juliette de Monaco was the illegitimate daughter of Marie Juliette Louvet, a cabaret singer, and Louis, Hereditary Prince of Monaco and Duke of Valentinois. She was adopted by Louis in 1919 and became heir presumptive to the throne as Hereditary Princess when her grandfather died and her father inherited the princely crown in 1922. Charlotte married Count Pierre de Polignac in 1920, but the marriage was not a happy one and they divorced in 1933. In 1944, Charlotte renounced and ceded her rights to the throne to her son Rainier. She died in 1977 in Paris. |\n| IS-Large-C Fewshot | Charlotte Louise Juliette de Monaco was the illegitimate daughter of Marie Juliette Louvet, a cabaret singer, and Louis, Hereditary Prince of Monaco and Duke of Valentinois, son and heir of Monaco's reigning monarch, Prince Albert I. In 1911, a law was passed recognising Charlotte as Louis's daughter, and declaring her to be a dynastic member of the sovereign family. In 1919, Louis adopted Charlotte in Paris, and she became heir presumptive to the throne as Hereditary Princess when her grandfather died and her father inherited the princely crown in 1922. Charlotte married Count Pierre de Polignac in 1920, but the marriage was not a happy one and the couple divorced in 1933. In 1944, Charlotte renounced and ceded her rights to the throne to her son Rainier, subject to the stipulation that he did not predecease her. Charlotte died in 1977 in Paris. |\n| GPT-3.5 Zeroshot | Charlotte was an illegitimate child of the heir to the throne of Monaco. In order to legitimize her and prevent a succession crisis, she was married off to a Frenchman who took the Grimaldi name. The couple had two children before divorcing. Charlotte later moved to Paris, where she died. |\n| GPT-3.5 Fewshot | Princess Charlotte of Monaco was the illegitimate daughter of Marie Juliette Louvet, a cabaret singer, and Louis, Hereditary Prince of Monaco and Duke of Valentinois, son and heir of Monaco's reigning monarch, Prince Albert I. Louis had no legitimate children or siblings, so Charlotte was adopted into the dynasty in order to prevent a succession crisis. She married Count Pierre de Polignac of Hennebont, Morbihan, Brittany, France, who took the surname Grimaldi and became a Prince of the Monegasque Royal Family. The couple had two children, but their marriage was not a happy one and they divorced in 1933. Princess Charlotte renounced her rights to the throne in 1944, and she died in 1977. |\n| Human | Princess Charlotte of Monaco, Duchess of Valentinois (Charlotte Louise Juliette Grimaldi; 30 September 1898 - 16 November 1977), was the daughter of Louis II, Prince of Monaco, and the mother of Rainier III, Prince of Monaco. From 1922 until 1944, she was the Hereditary Princess of Monaco, heir presumptive to the throne. |\n\n\n\n\n\n\n| System | Summary |\n| --- | --- |\n| IS-Large-C Zeroshot | The article is discussing a pedal assembly that is used to control the throttle of a vehicle. The assembly includes a mounting bracket, a guide structure, a carrier assembly, a drive assembly, a pedal assembly, a resistance assembly, and a generator. The pedal assembly is adjustable to accommodate operators of varying anatomical dimensions, and the resistance assembly provides damping for a non-adjustable pedal assembly. |\n| IS-Large-C Fewshot | The invention accelerator pedal assembly 10 includes a mounting bracket 11, a guide structure 12, a carrier assembly 13, a drive assembly 14, a pedal assembly 16, a resistance assembly 18, and a generator means 20. The pedal assembly 10 is intended to allow efficient fore and aft movement of the pedal assembly to accommodate operators of varying anatomical dimension and is operative to generate an electronic or drive - by - wire signal in response to pivotal movement of the pedal assembly while retaining the same ergometric operation of the pedal irrespective of the position of adjustment of the pedal. |\n| GPT-3.5 Zeroshot | The invention pedal assembly provides the desired adjustability, the desired ergometrics, and the desired hysteresis effect in a structure that is simple, inexpensive, and positive and reliable in operation. |\n| GPT-3.5 Fewshot | The invention provides a pedal assembly for a motor vehicle that can be adjusted to accommodate operators of varying anatomical dimensions, that has constant ergometrics irrespective of the position of adjustment of the pedal structure, and that provides a desired hysteresis effect in any position of adjustment of the pedal structure. The pedal assembly is simple, inexpensive, and positive and reliable in operation. |\n| Human | an electronic adjustable control pedal assembly for a motor vehicle including a carrier , a guide rod adapted to be secured to the dash panel of the vehicle and mounting the carrier for fore and aft movement along the guide rod , a power drive operative to move the carrier along the guide rod , a pedal arm pivotally mounted on the carrier , a potentiometer mounted on the carrier and operative to generate an output electrical signal proportioned to the extent of pivotal movement of the pedal arm , and a resistance assembly to provide feedback or feel to the operator . the resistance assembly includes an annular friction surface defined on the carrier , a sleeve positioned over the friction surface , and a coil spring encircling the sleeve and arranged to be tightened in response to pivotal movement of the pedal arm to squeeze the sleeve against the friction surface and generate a friction resistance force . the friction resistance force adds to the torsional resistance force of the spring during application of the pedal and subtracts from the torsional resistance force of the spring upon release of the pedal , whereby to create a hysteresis effect . |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper introduces INHERITSUMM, a versatile, compact summarization model distilled from GPT-3.5, displaying strong zeroshot and fewshot summarization potential similar to GPT-3.5 while being more compact.\n2. A commendable effort was made to gather a large and diverse summarization dataset.\n3. The use of ROUGE scores in experiments underscores the model's effectiveness.", "reasons_to_reject": "1. While the abstract mentions the potential shortcomings of ROUGE scores, the experiments exclusively rely on them. This makes it uncertain if InheritSumm genuinely surpasses the baselines. The absence of case studies also limits insights into the model's performance.\n2. The paper doesn't specify if the dataset or the model will be shared, and the lack of content in the appendix raises concerns about reproducibility, crucial for empirical research.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper introduces INHERITSUMM, a distilled, versatile model from GPT-3.5, addressing the challenges of large model fine-tuning costs while retaining robust zeroshot and fewshot summarization performance. Experimental results highlight its prowess.", "excitement": 3, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents a large curated dataset GPTSumm, collected using documents from Pile as the source and GPT3.5 summaries. The paper studies the performance of a summarization model distilled from this dataset. Both these are valuable contributions for future work on model distillation for summarization. The evaluation of the model is missing any human evaluation or discussion/analysis of factuality of generated summaries (standard now for summarization papers); these will strengthen the claims in the paper.
Decision: Accept-Findings