DIONYSUS: A Pre-trained Model for Low-Resource Dialogue Summarization
Year: 2023

Abstract:
Dialogue summarization has recently garnered significant attention due to its wide range of applications. However, existing methods for summarizing dialogues have limitations because they do not take into account the inherent structure of dialogue and rely heavily on labeled data, which can lead to poor performance in new domains. In this work, we propose DIONYSUS (dynamic input optimization in pre-training for dialogue summarization), a pre-trained encoder-decoder model for summarizing dialogues in any new domain. To pretrain DIONYSUS, we create two pseudo summaries for each dialogue example: one from a fine-tuned summarization model and the other from important dialogue turns. We then choose one of these pseudo summaries based on information distribution differences in different types of dialogues. This selected pseudo summary serves as the objective for pre-training DIONYSUS using a self-supervised approach on a large dialogue corpus. Our experiments show that DIONYSUS outperforms existing methods on six datasets, as demonstrated by its ROUGE scores in zero-shot and few-shot settings.

heading: 1 Introduction
text: Text summarization aims to produce concise and accurate summaries of long texts. Recent research on pre-trained neural language models has shown success in summarizing monologues (Lewis et al., 2020; Raffel et al., 2022; Zhang et al., 2019; He et al., 2022), such as news articles (Lee et al., 2022; Ravaut et al., 2022) and scientific publications (Ibrahim Altmami and El Bachir Menai, 2022; Dong et al., 2021). However, dialogue summarization presents additional challenges due to the different information distribution in dialogues.
Self-supervised text summarization models (Zhang et al., 2019; Wan and Bansal, 2022; Phang
∗Work was done when Yu Li was interning at MSR
et al., 2022) are typically pre-trained on free-form text data, with selected sentences as the pre-training objective. While this approach can be effective for monologues such as news articles, it is less successful at summarizing semistructured and multiparticipant dialogues. As illustrated in Figure 1, in daily chats, dialogue information is often dispersed across various dialogue turns, making it difficult to extract all relevant information through a few selected turns. While a golden summary needs to accurately captures vital information throughout the entire conversation. Furthermore, real-world dialogue-summarization applications often have limited or even no labeled data, making it challenging to develop effective models. Therefore, it is crucial to develop dialogue summarization models that can perform well in zero-shot and few-shot
1368
settings for their practical use. To address these challenges, we propose DIONYSUS, a pre-trained sequence-to-sequence model designed to summarize dialogues in any domain, even with a lack of labeled data. It uses pseudo summaries as its pre-training objective, which can be dynamically selected from two sources.
First, for daily chats where multiple dialogue turns are not sufficient to summarize the dialogue, we train a summary helper using high-quality dialogue summarization datasets to generate pseudo summaries for these types of dialogues. On the other hand, for dialogues like meeting minutes, interviews, and debates, which can be summarized through a selection of essential turns, we use a method inspired by the gap sentence generation (GSG) technique in PEGASUS to select these turns as pseudo summaries for training. For instance, choosing the final few turns in a conversation can effectively summarize meeting minutes. We have improved upon the GSG method by using the generated summaries from the summary helper as references during gap sentence selection, as they tend to have less noise compared to the full dialogue context. We refer to this source of pseudo summaries as “Principal” and refer to our improved method as GSG+. We find that our improved method outperforms previous methods in low-resource settings across different domains, such as daily chats, emails, and customer service dialogues. Additionally, we study different objective strategies for selecting the pseudo summary as a pre-training objective from the generated summary and the “Principal.”
We evaluate DIONYSUS on six dialogue summarization datasets. Our best model trained on 19 dialogue corpora surpasses PEGASUSLARGE in a zero-shot setting across all domains. We also found that the best performance is achieved by selecting the source with the highest ROUGE score as the objective strategy. Our main contributions are:
• The development of DIONYSUS, a pretrained sequence-to-sequence model for summarizing dialogues in any domain in a zeroshot or few-shot setting.
• The introduction of new self-supervised pretraining objectives for dialogue summarization using a summary helper and GSG+.
• The demonstration that DIONYSUS outperforms baselines on six domains in low-
resource settings, and can be fine-tuned with only 10 training examples to outperform vanilla T5 (Raffel et al., 2022) fine-tuning with 1, 000 examples.

heading: 2 Approach
text: Figure 2 outlines the steps for constructing DIONYSUS: § 2.1 First, a summary helper is constructed using two high-quality dialogue summarization datasets. This helper generates a pseudo summary for each dialogue in our pre-training corpus. § 2.2 Next, the “Principal” is extracted using GSG+ as the other pseudo summary for the dialogue. § 2.3 Finally, various strategies are employed to select the best pseudo summaries from the first and second steps to serve as the objective for pre-training.

heading: 2.1 Summary Helper
text: In certain types of dialogue, such as daily chats, it can be challenging to gather all necessary information from just a few dialogue turns due to the dispersed nature of dialogue information. To address this problem, we have created a summary helper model that generates pseudo summaries for each training example in our pre-training corpus.
We build our summary helper upon the T5 (Raffel et al., 2022) model. To capture essential information in a dialogue, we have trained our helper on the MultiWoz dataset (Budzianowski et al., 2018; Eric et al., 2020) in DS2 (Shin et al., 2022), which contains summaries derived from dialogue states using templates. This allows us to capture essential information from each turn in the conversation. Additionally, we have continued training our helper on the DialogSum (Chen et al., 2021) dataset, a human-annotated dataset in the daily life domain. This allows us to overcome the fixed format of summaries introduced by templates in DS2 and produce more natural pseudo summaries.

heading: 2.2 Gap Sentence Generation Plus (GSG+)
text: Algorithm 1 GSG+ 1: P ← ∅ 2: for j ← 1 to m do 3: si := rouge(P ∪ {xi}, G),∀i s.t. xi /∈ P 4: k := argmax{si}n 5: P := P ∪ {xk} 6: end for
Dialogues in certain settings, such as meetings and medical dialogues, often include summary
turns that summarize the entire conversation. For example, a participant may summarize a meeting, or a doctor may explain the outcome. These summary turns can be used as a pre-training objective because they highlight the main points of the dialogue and provide a concise overview of the topic discussed. In order to make DIONYSUS more adaptable to these scenarios, we have improved the independent principal method in the GSG method (Zhang et al., 2019) by using it to select essential summary turns as pseudo summaries for training. Our new method, called Gap Sentence Selection Plus (GSG+), uses the ROUGE1-F1 score between each dialogue turn xi and the generated summary G from the helper in Section 2.1 rather than the remaining text D \ xi to determine the importance of each turn. The generated summary eliminates much of the extraneous information from the dialogue and thus tends to have less noise than the full dialogue context, resulting in a less cluttered summary. This enables us to select the top-m-scored summary turns as the “Principal,” which will provide a more comprehensive overview of the vital information in the dialogue. For instance, Using the summary helper to identify key points increases the likelihood of selecting the most important dialogue turns as the “Principal” summary when creating pseudo summaries for meeting minutes instead of randomly selecting dialogue turns.
Specifically, given a dialogue D = {xi}n, we use Algorithm 1 to obtain the pseudo-summary “Principal” P . The input for our training example is the remainder of the dialogue D \ P . In Appendix C, we explore the impact of the dialogue turns order on the formation of the “Principal”. Using GSG+ can effectively identify essential summary turns and generate more accurate pseudo-summaries than with the original GSG method.
Algorithm 2 Better ROUGE 1: S ← ∅ 2: sg := rouge(G,D \ {P}) 3: sp := rouge(P,D \ {P}) 4: if sg > sp then 5: S := G 6: else 7: S := P 8: end if

heading: 2.3 Pre-training Objectives Strategy
text: To generate the final pseudo summary S for each specific dialogue training example, we consider three strategies. These strategies are based on the generated pseudo summary G and the extracted “Principal” P . These strategies serve as the pretrain objective for the dialogue training example.
All G S = G: We always select the generated summary from the summary helper as the pretraining objective.
All P S = P : We always select the “Principal” as the pre-training objective.
Better ROUGE We use either G or P based on the recall of information from the dialogue to determine the pre-training objective. We utilize Algorithm 2 to get the pre-training objective by calculating the ROUGE1-F1 score for the pseudo summaries and the dialogue, excluding the “Principal” D \ P . It is important to note that we use the same reference to ensure a fair comparison.
For pre-training with above strategies, if we choose G as the pseudo summary, we input the full dialogue. If we choose P , we input the dialogue, excluding the “Principal,” D \ P to create an abstract summary. However, we also include the “Principal” with a probability, using a copying mechanism to create an extractive summary.
More information about this copy mechanism can be found in Section 5.4. It is important to note that we do not combine these two pseudo summaries for a single training example. Each example in our pre-training corpus will have either G or P as its designated pseudo summary.

heading: 3 Training Corpus
text: To train DIONYSUS, we utilized 19 conversational corpora that do not come with pre-defined dialogue summaries. We employed a self-supervised approach by using pseudo-summaries as the pretraining objective.
Conversational Corpora We collect 19 available conversational corpora consisting of 1.7M examples after truncating for pre-training. Corpus information is listed in Table 1. We access these corpora through ConvoKit v2.5.31. This helps us to ensure that DIONYSUS is well-equipped to handle a variety of conversational scenarios.
We train our objective summary helper with a rule-based dialogue summarization dataset (DS2) and an abstractive summarization dataset (DialogSum).
1https://convokit.cornell.edu/ 2Cornell Movie Dialogs Corpus is from Danescu-
Niculescu-Mizil and Lee (2011) 3PERSUASIONFORGOOD is from Wang et al. (2019) 4https://convokit.cornell.edu/documentation/redditsmall.html 5https://convokit.cornell.edu/documentation/supreme.html 6Wikipedia Talk Pages is from Danescu-Niculescu-Mizil et al. (2012)
DS2 This dataset (Shin et al., 2022) creates dialogue summaries for the MultiWOZ (Budzianowski et al., 2018; Eric et al., 2020) dataset by heuristic rules from the dialogue states. It includes 5 domains and 10, 000 dialogues.
DialogSum This dataset (Chen et al., 2021) collects human annotated summaries for daily-life dialogues from three datasets: DailyDialog (Li et al., 2017), DREAM (Sun et al., 2019), and MuTual (Cui et al., 2020), as well as dialogues from an English-speaking practice website. It has 13,460 dialogues in total.

heading: 4 Experiments

heading: 4.1 Downstream Tasks and Metrics
text: We evaluate our methods on three public dialogue summarization datasets or benchmarks: SAMSum (Gliwa et al., 2019), ConvoSumm (Fabbri et al., 2021), and TWEETSUMM (Feigenblat et al., 2021)
SAMSum This dataset contains over 16k natural messenger-like dialogues with manually annotated summaries by language experts.
ConvoSumm It is a benchmark of four domains: New York Times comment, StackExchange, W3C email, and Reddit. Dialogues are extracted from publicly available data, and each domain has 500 dialogues. They hire crowdsorce workers on Amazon Mechanical Turk to annotate dialogue summary.
TweetSumm This dataset contains 1,100 reconstructed real-world customer support dialogues from Tweet. Each dialogue has human annotated abstractive summaries and extractive summaries. We only use abstractive summaries in the dataset as references in our experiments.
We report ROUGE-1, ROUGE-2, and ROUGEL scores (Lin, 2004) to evaluate generated summaries against references.

heading: 4.2 Baselines
text: We compare our methods with three competitive baselines.
T5v1.1 It is an improved version of the original T5 model (Raffel et al., 2022). Since the original T5 model is pre-trained on downstream tasks in supervised learning, the test set of downstream tasks overlaps with the pre-training data. To make a fair comparison in a zero-shot setting, we choose T5v1.1 as it is pre-trained on C4 without mixing in the downstream tasks.
PEGASUS Zhang et al. (2019) propose this pretrained model for abstractive summarization tasks. The pre-training objective is GSG, transforms any text into an abstractive summarization example by selecting important sentences as output summaries. We use the PEGASUSLARGE checkpoint7 as there is no publicly available PEGASUSBASE checkpoint.
GSG* We use the independent principal strategy of GSG training objective in PEGASUS (Zhang et al., 2019) but pre-train DIONYSUS with our training corpora. We build this baseline to explore the performance gap between our pre-training objective and GSG.

heading: 5 Results and Analysis
text: We focus on low-resource dialogue summarization settings because it is difficult to collect enough training examples. We evaluate DIONYSUS with “All G”, “All P”, and “Better ROUGE” strategies in zero-shot and few-shot settings and compare it to the baselines.

heading: 5.1 Zero-Shot Results
text: In order to evaluate the effectiveness of DIONYSUS, we conduct a zero-shot test on DIONYSUSLARGE with all strategies and other baselines. We present the results in Table 2. The ROUGE1-F1, ROUGE2-F1, and ROUGEL-F1 scores are used as the standard evaluation measures for summarization tasks. Our models show impressive performance improvements over the baselines on all downstream datasets. Specifically, DIONYSUSLARGE with the “Better ROUGE” strategy performs the best overall across all downstream datasets (Average: ROUGE-1/2/L: 29.7/8.0/20.2), indicating that it benefits from both generated and extractive pseudo summaries and can adapt to various domains. The “All P” strategy performs better than the GSG* baseline on most datasets, indicating that our Gap Sentence Selection Plus method can effectively select dialogue turns that provide an accurate dialogue summary. Additionally, the DIONYSUSLARGE with “All G” and “Better ROUGE” strategies demonstrate significant improvement compared to T5v1.1LARGE (Average ROUGE2: +5.6/ + 6.1) and PEGASUSLARGE (Average ROUGE2: +2.2/ + 2.7), indicating that pre-training with our summary helper is
7https://huggingface.co/google/pegasus-large
highly beneficial. However, the “All G” strategy only performs as well as the “Better ROUGE” strategy on the SAMSum dataset (ROUGE-1/2/L/: 41.3/16.1/30.6 → 41.3/16.2/30.9), suggesting that the improvement from the summary helper is more pronounced on this particular dataset. This may be due to the similarity between the datasets used to train the helper and the SAMSum dataset, which we discuss further in Sections 5.5 and 5.6. Overall, our models outperform previous methods, such as PEGASUS, in a zero-shot setting, demonstrating their effectiveness and potential for further development.

heading: 5.2 Few-Shot Results
text: We investigated reducing annotation labor in dialogue summarization tasks by using few-shot dialogue summarization. We report ROUGE1-F1, ROUGE2-F1, ROUGEL-F1, and ROUGELSumF1 scores to evaluate model performance. Specifically, We fine-tune DIONYSUSLARGE, PEGASUSLARGE, and T5v1.1LARGE with the first 1/10/100/1K/10K training examples from the SAMSum dataset. We show the results of our experiments with varying training data sizes in Figure 3. We found that all models improved with more examples. Among these models, DIONYSUSLARGE consistently outperformes both PEGASUSLARGE and T5v1.1LARGE when trained with a dataset ranging from 0 to 10, 000 examples. This suggests that our pre-training process helps DIONYSUS adapt to downstream tasks more quickly. Additionally, we observed that PEGASUSLARGE outperformed T5v1.1LARGE due to its pre-training on summarization tasks. Figure 3 shows the gap between DIONYSUSLARGE and PEGASUSLARGE is particularly significant when using fewer than 100 training examples, indicating better recall capabilities in dialogue summarization for DIONYSUS. Even with only 10 training examples, DIONYSUSLARGE achieves higher ROUGE scores than the T5v1.1LARGE model trained with 1,000 examples, making it the best option for lowresource dialogue summarization.

heading: 5.3 Effect of Compression Ratio
text: In GSG+, we can choose a fixed number of turns in the dialogue as a training objective or select turns with a compression ratio. We investigate the compression ratio in a dialogue turn level as the number of selected turns over the number of totals turns in the dialogue (Nprincipal/Ndialogue). A
low compression ratio will select fewer turns in the dialogue as the objective, making pre-training less challenging. However, it tends to have a lower ROUGE1-F1 score with the remaining dialogue turns, meaning the “Better ROUGE” strategy selects more generated summaries as the objective. While choosing a high compression ratio will make the pre-training more challenging. Nevertheless, it has a higher ROUGE score compared to generated summaries, leading to more principal under the “Better ROUGE” strategy. We show the zero-shot
performance on development sets of the SAMSum dataset and TweetSumm dataset with compression rates from 10% to 60% in Figure 4. It shows that the model with 15% compression ratio achieves the highest ROUGE-2 score.

heading: 5.4 Effect of Copying Mechanism
text: The copying mechanism is important for dialogues like meetings and medical dialogues because it allows for summarization of entire dialogue through several turns. As shown in Table 3, we compare the performance of the “All P” strategy to a scenario where 50% of the selected dialogue turns are retained in the input rather than being removed. In this case, the input for each pre-training example includes the entire dialogue D, rather than D \ P . This leads the model to focus on extractive summarization. We observed that adding a random copy mechanism significantly improved the overall performance. Additionally, we
also evaluate the “Better ROUGE” strategy with different copying probabilities ranging from 0.15 to 0.7. In these experiments, we choose top-2 dialogue turns as principal, which results in 51.9% of pre-training objectives being the principal, and the rest is the generated summary. Figure 5 shows that leaving 15% of dialogue turns in the principal best enhances the overall quality of dialogue summarization.

heading: 5.5 Comparison Between All G and Summary Helper
text: Since the summary helper model provides the generated summary as an objective candidate and has shown strong capabilities in zero-shot dialogue summarization. As shown in Table 4, we compare the helper model to our “All G” model in a zeroshot setting. The difference is that we train the “All G” model on the pre-training corpora annotated by the helper. We found that the helper model is not on par with our model. While the helper model may have performed well on a particular task (NYT), its overall performance is not as strong as our model. This is because DIONYSUS has been extensively trained on various dialogue datasets, which makes it consistently perform well in a wide range of tasks
and scenarios.

heading: 5.6 Test-Set Overlap with Pre-Training Corpora
text: In order to ensure a fair comparison, we check for overlap between pre-training and downstream test datasets. This is done by calculating the similarity between all pairs of test set targets in the SAMSum dataset and pre-training documents using the ROUGE2-recall measure, which is calculated as the number of overlapping bigrams divided by the total number of bigrams in the test target. We then count the number of test set examples that have a similarity to any pre-training example above a certain threshold. As shown in Table 5, the overlap between the SAMSum dataset and the datasets used for training the helper and the pre-training datasets is low when the similarity threshold is set between 0.4 and 1.0. This suggests that there is not significant similarity between our test set and the pre-training datasets. It indicates that the improvement in DIONYSUS is due to the pre-training process rather than potential test data leakage.

heading: 5.7 Human Evaluation
text: We evaluate the performance of DIONYSUS by conducting human evaluation experiments on Amazon Mechanical Turk. We randomly select 100 examples from the SAMSum dataset to compare summaries generated by our model with those written by humans in the dataset. We choose DIONYSUS trained with the “Better ROUGE” strategy and
generate summaries in a zero-shot setting. Participants are asked to rate the summaries on a scale of 1 to 5, with higher scores indicating better quality. We collect the scores from three participants for each example and report the average scores in Table 6. A paired t-test is conducted to determine if scores are significantly different between our model and other models. Our results show that DIONYSUS could generate summaries of similar quality to human-written summaries without any training data. DIONYSUS also gets better ratings than the vanilla T5 and PEGASUS models, which aligns with the results obtained from the automatic evaluation. More information on the human evaluation process can be found in Appendix F.

heading: 6 Related Work
text: Dialogue summarization is a rapidly growing area of research that focuses on automatically generating concise and informative summaries of conversations (Feng et al., 2022). Unlike research on traditional documents like news articles (Fabbri et al., 2019; Ahuja et al., 2022) or scientific papers (Lu et al., 2020; Ibrahim Altmami and El Bachir Menai, 2022), dialogue summarization is particularly relevant in multi-party interactions, such as emails (Zhang et al., 2021), meetings (Carletta et al., 2005), medical dialogues (Zeng et al., 2020), and daily chats (Chen et al., 2021). However, many existing methods for dialogue summarization require a large training dataset with annotated summaries. This can be a major barrier to applying these methods in real-world scenarios, particularly in cases with limited or no annotated data available. Our study examines the use of dialogue summarization in low-resource settings to make the process more practical and effortless in various contexts.
Pre-trained Transformer-based (Vaswani et al., 2017) language models (Devlin et al., 2019; Radford et al., 2019; Yang et al., 2019) have become increasingly popular in natural language processing tasks for tackling the data shortage problem. However, many of these models have limitations when it comes to dialogue summarization. Zhang et al. (2019) propose PEGASUS, which masks multiple whole sentences and pre-trains sequence-tosequence models to reconstruct the original text. Built on that, Wan and Bansal (2022) improve the sentence selection strategy and add modules for ensuring factuality during fine-tuning to address the problem of factuality in summarization. Phang
et al. (2022) extend PEGASUS with a modified architecture and long-sequence pre-training to tackle long-input summarization. He et al. (2022) propose ZCode++, a pre-trained language model optimized for abstractive summarization with improved encoder. However, all these methods rely on the Gap Sentence Selection method, which has limitations for dialogue summarization. In contrast, our approach uses pseudo-summary construction as the pre-training objective, making it possible for zeroshot dialogue summarization.
Another line of work focuses on pre-trained models for dialogues. DialoGPT (Zhang et al., 2020) and PLATO (Bao et al., 2020), which are pretrained on large-scale conversation datasets such as Reddit. For dialogue summarization, Jia et al. (2022) post-train pre-trained language models to rephrase dialogues into narratives and then finetunes them for summarization. In contrast, our approach follows the T5 model’s unified text-to-text format for both pre-training and fine-tuning. Zhong et al. (2022) train UNILM (Dong et al., 2019) with a window-based denoising framework for long dialogue understanding and summarization but do not focus on low-resource settings. Zou et al. (2021) propose a pre-training paradigm that pre-trains the encoder and decoder separately in a supervised manner. While our method uses a self-supervised pre-training approach that applies to any dialogue dataset, making it easier to extend to larger pretraining corpora for further improvement.

heading: 7 Conclusion and Future Work
text: We present DIONYSUS, a pre-trained encoderdecoder model for zero-shot dialogue summarization in any new domain. We pre-train using a self-supervised approach that generates pseudosummaries for large dialogue corpora as the pretraining objective. We investigate the impact of various pre-training objective strategies and model sizes on dialogue summarization performance. Our experiments show that DIONYSUS outperforms state-of-the-art models on six datasets in a zeroshot setting. Furthermore, DIONYSUS can be fine-tuned with only 10 examples to outperform vanilla T5 fine-tuning with 1,000 examples. This makes dialogue summarization more practical and easier to use in various contexts with minimal effort. We plan to extend this method to abstractive summarization tasks to develop a general zero-shot summarization model.

heading: 8 Limitations
text: Training Data Our pre-training data is sourced from 19 existing dialogue datasets. However, it’s important to note that these datasets may contain noise, such as harmful content, irrelevant file names, and URL links. Despite utilizing multiple automatic tools to filter out this content during preprocessing, there is still a chance that some noise may be present in our pre-training data. This could potentially impact the performance of DIONYSUS, making it important to monitor and improve the pre-processing steps continuously.
We also know the potential drawbacks of constructing pseudo summaries using the GSG method, which may lead to unnatural summaries for dialogue data. To mitigate this, we introduced the Summary Helper in Section 2.1, which is specifically trained on two dialogue summarization datasets containing natural summaries. This approach enables more realistic pseudo-summaries and enhances zero-shot performance. Although we employ top-m turns as an additional source of pseudo summaries, Figure 4 illustrates that GSG+ contributes a minor portion of the pseudo summary, with a 0.7 to 0.3 ratio between generated and topm turns. Our method thus minimizes referent and pronoun confusion, ensuring better coherence than solely employing the standard GSG technique.
Training Resource To improve our model’s performance, we employ the “Better ROUGE” strategy, which calculates the ROUGE score for both candidates and selects the best one as the final training objective. This data pre-processing process can be pretty time-consuming, taking approximately one day to complete for our pre-training data when utilizing 100 threads. Additionally, we utilize 16 Nvidia V100 GPUs to train our models, which may not be accessible or reproducible for all researchers. This could present a significant obstacle for those looking to replicate or build upon our work.
Test Data Another potential concern is the test datasets used to evaluate DIONYSUS. The test set size is relatively small, which may not fully represent the breadth of dialogue types that a general dialogue summarization model should be able to handle. This could lead to the model performing well on the test set but not generalizing to other unseen dialogue types. Further, our analysis did not include the assessment of long dialogue summarization, such as lengthy meetings (Carletta et al., 2005;
Zhong et al., 2021; Janin et al., 2003) or screenplays (Chen et al., 2022). However, our study’s approach has the potential to handle these scenarios, even though it was not specifically designed for them. By incorporating LongT5 (Guo et al., 2022) or DialogLM (Zhong et al., 2022), which are known for their ability to process extended input sequences, we expect that they could efficiently tackle this task.

heading: 9 Acknowledgement
text: Our gratitude goes out to Microsoft Research for providing us with computational resources. We would also like to thank Kun Qian for valuable discussions and the Columbia NLP and Microsoft Deep Learning Group members for their feedback and discussions. Additionally, we thank the Mechanical Turk workers for conducting the human evaluation.

heading: B Additional Base Model Results
text: Table 7 presents the results of DIONYSUSBASE in a zero-shot setting, and Figure 6 compares the few-shot results of DIONYSUSBASE with those of the T5 base model. These initial results demonstrate the potential for further analysis and optimization of DIONYSUS. Upon comparison with other baselines, it is clear that DIONYSUS performs better under both zero-shot and few-shot conditions, outperforming the GSG* model. These results provide valuable insight into the capabilities of DIONYSUS and can inform the development of larger models.

heading: C Effect of the Dialogue Turns Order in Principal
text: We could use two possible orders to align the dialogue turns in the principal. The first order is to align the text with the ROUGE1-F1 score. The second order is to align the principal with the order in the original dialogue. This means that the principal will be arranged in the same order as in the original dialogue, without rearrangement. This option helps preserve the original flow and structure of the dialogue. We compare these two orders of principal in the GSG* baseline. As shown in Table 8, the results suggest that keeping the order in the original dialogue helps improve zero-shot performance as it provides a more nuanced understanding of the dialogue. We choose this order for all our models.

heading: D Pre-training Steps
text: To evaluate the performance of DIONYSUS during pre-training, we measured the ROUGE1-F1, ROUGE2-F1, ROUGEL-F1, and ROUGELSumF1 scores on the SAMSum dataset in Figure 7. We keep track of the model’s progress by logging its performance every 1,000 training steps. This allows us to monitor the model’s improvements over time and confirm that it is learning effectively.

heading: E Example Model Outputs
text: In order to evaluate the performance of DIONYSUS, we randomly selected model output examples
from both the SAMSum dataset and the TweetSumm dataset. We report these examples with their corresponding gold summaries in Tables 9 and 10. The gold summaries served as a benchmark for our model’s output, allowing us to compare and estimate the quality of the generated summaries. We found that DIONYSUS could generate zero-shot summaries on par with those written by humans. However, we also identified factual errors in the generated summaries, such as misunderstandings of the subject matter. These errors suggest room for improvement in DIONYSUS, and we plan to address this issue in future work.

heading: F Human Evaluation Details
text: In our human evaluation experiments, we utilized the task template shown in Figure 8. Mechanical workers were instructed to rate four summaries for a given dialogue on a scale of 1 (poor) to 5 (excellent). To minimize bias, we provided a di-
alogue with its corresponding gold summary as an example of a high-quality summary. The summaries were presented in a randomized order for each task to prevent order bias. Three different workers independently completed each task, and the median score across all workers was retained for each summary. Participants were compensated with 0.3 USD per task, and we implemented the following qualifications for worker selection to ensure a high level of quality: (1) HIT approval rate for all requesters’ HITs is greater than 90%. (2) Location is one of AU, NZ, GB, and US. (3) Number of HITs approved is greater than 100.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Section 8

heading: 3 A2. Did you discuss any potential risks of your work?
text: Section 8

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Abstract and Section 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Section 1

heading: 3 B1. Did you cite the creators of artifacts you used?
text: Section 1

heading: 3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts?
text: Appendix B

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Section 1
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. It is discussed in the original artifacts I use.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Not applicable. It is discussed in the original artifacts I use.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Appendix A
C 3 Did you run computational experiments? Section 5

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Appendix A
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Appendix A

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 5

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Appendix A
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? Section 5.7

heading: 3 D1. Did you report the full text of instructions given to participants, including e.g., screenshots,
text: disclaimers of any risks to participants or annotators, etc.? Appendix F

heading: 3 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students)
text: and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Appendix F

heading: 3 D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to
text: crowdworkers explain how the data would be used? Appendix F
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Not applicable. It is in the Amazon Mechanical Turk user agreement protocal.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Not applicable. It is in the Amazon Mechanical Turk user agreement protocal.

heading: Future Work
text: These errors suggest room for improvement in DIONYSUS, and we plan to address this issue in future work