COMPOsitional Data Augmentation for Abstractive Conversation Summarization
Year: 2023

Abstract:
Recent abstractive conversation summarization systems generally rely on large-scale datasets with annotated summaries. However, collecting and annotating these conversations can be a time-consuming and labor-intensive task. To address this issue, in this work, we present a sub-structure level compositional data augmentation method, COMPO, for generating diverse and high-quality pairs of conversations and summaries. Specifically, COMPO first extracts conversation structures like topic splits and action triples as basic units. Then we organize these semantically meaningful conversation snippets compositionally to create new training instances. Additionally, we explore noise-tolerant settings in both self-training and joint-training paradigms to make the most of these augmented samples. Our experiments on benchmark datasets, SAMSum and DialogSum, show that COMPO substantially outperforms prior baseline methods by achieving a nearly 10% increase of ROUGE scores with limited data. We have publically released our code at https://github.com/ ozyyshr/Compo.

heading: 1 Introduction
text: Abstractive conversation summarization, which condenses unstructured conversations into short, concise, and structured text, has greatly benefited from neural generative models trained on largescale annotated data. Researchers have focused on various aspects in conversation summarization, such as hierarchical modeling of conversations (Zhao et al., 2019; Zhu et al., 2020), leveraging dialogue acts (Goo and Chen, 2018), using key phrases and entities (Liu et al., 2019a; Narayan et al., 2021), utilizing topic segments (Liu et al., 2019b), incorporating stage components (Chen and Yang, 2020) and examining discourse relations (Chen and Yang, 2021b; Feng et al., 2020b). However, training these generative models often requires abundant high-
quality data, i.e., conversation and its paired summary, which is usually time-consuming and laborintensive to obtain. As a result, it is challenging to apply them to generalized real-world situations where labeled summaries are limited.
A direct solution is to employ data augmentation (DA) (Cubuk et al., 2018; Sennrich et al., 2015; Feng et al., 2021a; Chen et al., 2021a,b; Shen et al., 2020; Yu et al., 2018; Feng et al., 2020a; Miyato et al., 2016) to generate more data. Whereas, directly applying these augmentation methods into the context of conversations usually fails to consider any unique structures of conversations such as speaker information, topic split, and conversation stages (Gritta et al., 2021; Shuster et al.,
1471
2021), which distinguish conversations from general sentences. As a result, they might be limited in creating high-quality and diverse data pairs (Chen and Yang, 2021a). Even though there are a few exceptions (Chen and Yang, 2021a; Liu et al., 2022), they still suffer from diversity and struggle with out-of-distribution compositional generalization (Feng et al., 2021a). One way to alleviate these issues is to recombine different data points to produce novel training data, i.e., compositional data augmentation (Akyürek et al., 2020; Zhang et al., 2022). However, existing compositional DA mainly focus on editing short sentences locally with words/phrases/parsing trees (Akyürek et al., 2020; Zhang et al., 2022), neglecting rich structural information between different sets of utterances in conversations (Chen and Yang, 2020; Cohan et al., 2018), which prevent them from being applied to conversations to compose multiple utterances and generate novel, diverse and high-quality conversational data.
We visualize one example with the topic structures (Xu et al., 2021; Galley et al., 2003; Chen and Yang, 2020) highlighted in Figure 1. The conversation consists of several topics: “opening”, “explanation”, “plan”, etc. And we consider every topic snippet as the basic unit. In the meantime , we extracted the “action” triples (Chen and Yang, 2021b) to represent each topic snippet. With these topic snippets and action representations, we obtain the units for compositional operations. For instance, the blue topic split and summary sentence about the meeting plan could be composed into another conversation by substitution to produce a new conversation and summary that contains a meeting plan. As it shows, by extracting the topic structures from the conversations, sub-components of conversations can be re-organized and re-composed to generate augmented conversation-summary pairs that might not be seen in the original corpus, resulting in more diverse training data.
To this end, we propose COMPO, a compositional data augmentation framework operating at sub-structure level. We leverage the conversation structures (i.e., topic structure(Chen and Yang, 2020) and action triples (Chen and Yang, 2021b) ) to produce compositional units for generating diverse conversation-summary pairs. Specifically, we first segment conversation into topic splits with topic modeling models, and then extract “actions triples” (Chen and Yang, 2021b) to represent each
split as actions express specific socially situated identities and activities. With the extracted structures, we view the topic snippets as the basic units and perform selective retrieval based on action triples for compositional substitution to generate novel and diverse conversations. We also pair topic splits with summary sentences so that new summaries would be generated as well. An example of newly augmented conversation and summary could be found in Figure 2(b). To better leverage the newly generated conversation-summary pairs from COMPO, we further explore two noisetolerant methods including a self-training framework that uses the new conversations only, and another joint-training framework that leverages paired data. Empirical studies verify COMPO’s effectiveness via both quantitative and qualitative evaluations on SAMSum (Gliwa et al., 2019) and DialogSum (Chen et al., 2021c) compared to prior state-of-the-art data augmentation techniques. We also illustrate COMPO’s transferability on a news summarization dataset CNN/Dailymail.

heading: 2 Related Work

heading: 2.1 Abstractive Conversation Summarization
text: Abstractive conversation summarization, as opposed to extraction summarization, requires generative models to have a strong ability in language understanding as the words in the output may not appear in the input. Prior work on abstractive conversation summarization can be divided into two categories. One is to directly apply existing document summarization models to conversations (Shang et al., 2018; Gliwa et al., 2019). The other is to design conversation-tailored methods, for instance, modeling conversations in a hierarchical way (Zhao et al., 2019; Zhu et al., 2020). The rich structured information in conversations has also been leveraged. For example, Goo and Chen (2018) used dialogue acts; Liu et al. (2019a); Narayan et al. (2021) leveraged key phrases and entities. Topic segments (Liu et al., 2019b), stage components (Chen and Yang, 2020) and discourse relations (Chen and Yang, 2021b; Feng et al., 2020b) are also explored to understand conversation context for summarization. However, most approaches in the aforementioned categories focus on neural supervised methods and require abundant data to achieve state-of-the-art performance, which is timeconsuming and labor-intensive. In this work, we introduce conversation-specific data augmenta-
tion methods to help address data scarcity on paired conversations and summaries.

heading: 2.2 Data Augmentation in NLP
text: Data augmentation (DA) is an effective approach to boost the performance of neural supervised models, and has been widely applied in various NLP tasks such as text classification (Wei and Zou, 2019; Zheng et al., 2020), machine reading comprehension (Yu et al., 2018), and machine translation (Sennrich et al., 2015). Commonly seen practices involve designed word/synonym replacement (Kobayashi, 2018; Niu and Bansal, 2018), word deletion/swapping/insertion (Wei and Zou, 2019), back translation (Sennrich et al., 2015; Xie et al., 2019) and compositional augmentation (Jia and Liang, 2016; Andreas, 2019).
However, it is not applicable to directly adopt general DA methods to conversations as they usually neglect conversation structure. By extending general DA methods, Liu et al. (2022) generates synthetic examples by replacing semantically similar text spans in both dialogue and summary. Chen and Yang (2021a) makes an initial attempt for structured conversational DA, but their approach could not guarantee compositional generalization, making it hard to create diverse augmentations. While compositional DA methods proved to be effective in solving the aforementioned issues, they often target plain text (Furrer et al., 2020) and operate locally with words, phrases, or parsing trees with carefully-curated rules (Chen et al., 2020b; Nye et al., 2020). Thus are not suitable for conversa-
tions. Our work COMPO fills these gaps by naturally taking conversation structures as units for compositional augmentation. In this way, we not only explore rich structures unique for conversations but also boost the compositional generalization and diversity.

heading: 3 Methodology
text: To generate diverse conversation-summary pairs to deal with the data scarcity issue, this section presents a simple and effective compositional data augmentation method COMPO for supervised abstractive conversation summarization. The framework is illustrated in Figure 2.

heading: 3.1 Compositional Augmentation
text: Our compositional augmentation method COMPO operates at the sub-structure level of conversations. By extracting different sub-components of conversations and recombining them based on certain orderings, COMPO can produce novel and diverse conversations and their summaries that might not be seen in the original corpus. To get a reasonable granularity of conversation sub-parts, we choose to leverage the topic-view of conversations, building upon prior work on conversation structures (Althoff et al., 2016; Chen and Yang, 2020). Conversations are mostly organized around topics in a coarse-grained structure (Honneth et al., 1988). For instance, a telephone chat could possess the following topics: greetings ⇒ invitation ⇒ plan ⇒ farewell. Thus we propose a compositional inductive approach through composing different
conversation topics (Andreas, 2019). We further employ COMPO to limited data settings in both self-training and joint learning styles.
Topical Split We employ the classic topic segmentation algorithm, C99 (Choi, 2000) to get the topical split of conversations based on intersentence similarities. First, we use Sentence-BERT (Reimers and Gurevych, 2019a) to get the representations for each utterance in the conversation C = {u1, u2, ..., um}. Then the conversation C is divided into blocks Ctopic = {b1, b2, ..., bn} with C99, where bi is one split topic block consisting of several consecutive utterances. Also, people tend to summarize conversations in an almost linear way with a strong temporal dependency (Wu et al., 2021). As a result, it is intuitive to pair each topical split Ctopic with summary sentences S = {s1, ..., sn} following Algorithm 1 to obtain sipaired for each bi.
Action Extraction Previous studies reveal that action information can be an effective building block for models to perform text generation (Daniel et al., 2003; Glavaš and Šnajder, 2014). Actions also help avoid less informative utterances in conversations such as dialog acts (Chen and Yang, 2021b), focusing on more concise ideas of conversation snippets. Therefore, we extract verb-centering phrases (Zhang et al., 2020a) as backbones of topic splits. We use a lightweight tool (Jiao et al., 2023; Zhong et al., 2022a) to extract the actions, where frequently-occurring syntactic patterns are leveraged. Specifically, we extract such syntactic patterns containing verbs as actions. For instance, the most common patterns contain n1-nsubj-v1 (e.g., Alice called). More details and concrete examples could be found in Appendix C.
Action-based Composition With previous steps, we obtain a pool P of topical splits and their corresponding actions P = {(bi, sipaired, Ai)}i=1:|p|. With these as units, we are now able to conduct compositional operations. To preserve the conversation structure of the augmented data, the general philosophy here is to “substitute” a selected conversation with similar candidates retrieved from the pool. The problem becomes how to filter out representative and diverse candidates. Inspired by Su et al. (2022), we use the graph-based method Vote-k to ensure similar demonstrations and total coverage. We first compute a vector representa-
tion for each topical unit using Sentence-BERT (Reimers and Gurevych, 2019b) by averaging the resulting vectors over the input. We then use those embeddings to create a directed graph G = (V,E). For each vertex v ∈ V , edges are connected to its k-nearest neighbors in terms of cosine similarity. For every remaining vertex u (contrary to chosen units) in the graph, we score them using
score(u) = ∑
v∈{v|(v,u)∈E,v∈U} s(v), (1)
where s(v) = ρ|{linL|(v,l)∈E}|, ρ > 1. In every iteration, we choose nodes that have the largest score, i.e., satisfy argmaxu∈Uscore(u). The chosen nodes are excluded from U .
In order to produce fluent conversations with newly generated compositional units, we leverage a pre-trained generation model. Concretely, we pre-train a sequence-to-sequence model in the following steps: (1) randomly select a topical split bi from the original conversation, (2) get the corresponding set of actions A = {a1, ..., ak} for bi, (3) mask bi from the original conversation, (4) take extracted actions A and unmasked the rest of the conversation as input. Then we use this selected topical split bi as the target output for the model. For example, the input and output of the pre-trained generation model could be
• Input: we ’ll meet at arrivals 
 Corina: Are youat the airport? <mask>
• Output: Regina: sure, waiting for K. Jorge: Good! we’ll meet at the arrivals then.
where “we ’ll meet at arrivals” is the combination of action triples, “
” is used to separate triples from conversations, and “<mask>” is what we want to predict as the output. If there are multiple actions, we use ‘|’ token to split them.

heading: 3.2 Noise-tolerant Training Settings
text: Our model is trained on two noise-tolerant settings to further boost the performance with limited data. In self-training setting, only the newly generated conversations are incorporated, and a teacher model is utilized to predict pseudo summaries. In joint-training setting, we test the framework with paired data, i.e., with newly generated conversations and summaries.
Algorithm 1: Match topical split and summary sentences
Input: A topical split of conversation bi ∈ Ctopic, a summary S containing n sentences, sliding window size interval [a,b] Output: Corresponding summary sentences Sipaired for bi
1 for w = a to b do 2 for j = 1 to |C| − w do 3 cand=Sj,j+w 4 r(j, w)← ROUGE(cand, bi) 5 W ←W ∪ cand 6 j ← j + w/2 7 w ← w + 1 8 jbest, wbest ← argmaxj,wr(j, w) 9 Sipaired ← Sjbest,(jbest+wbest)
Algorithm 2: Self-training 1 Train a base model fθ with labeled data
Dl = {(cli, si)}i=1:n 2 for i = 1 to K do 3 Predict pseudo summaries sui for unlabeled conversations Du = {(cui )}i=1:m 4 Select a subset of S = Dl ∪D where D = {cui , sui }i=1:a 5 train a new model fθ on S ∪Dl

heading: 3.2.1 Self-training with Augmented Data
text: The detailed algorithm for self-training (He et al., 2019) is displayed in Algorithm 2. Specifically, the algorithm starts with a parallel dataset Dl = {(cli, si)}i=1:n and the unlabeled dataset Du = {(cui )}i=1:m where m >> n. In a semi-supervised setting, a teacher model fθ is first trained on Dl, and is further used to predict pseudo summaries for unlabeled data. The pseudo data D and Dl are combined and we sample a subset of them for training another model fθ′ . Here θ is the parameter from the teacher model from the last iteration and fixed within the current iteration. This process is iterated for K times. The unsupervised loss Lu from unlabeled conversations is defined as:
Lu = −Ec∼DuEc′∼COMPO(c)logP (f(c; θ′)|f(c′; θ)) (2)
Note that we choose the number of subset selections so that the total training instances are twice the original dataset.

heading: 3.2.2 Joint Training with Augmented Pairs
text: Apart from using unlabeled conversations for selftraining, we can also generate pseudo summaries for augmented conversations, and perform joint training to see the effect.
New Summary Generation For each newly generated conversation, we leverage a pre-trained generation model similar to the model described in Section 3.1, and generate a new summary conditioned on summary context and the action triples.
Finally, the model is trained on a combination of the original samples and augmentation samples to obtain a trade-off between regularization and noise injection. The total training objective is:

heading: L = E(c,s)∈Dl logP (s|c)+γE(c′,s′)∈D′ logP (s′|c′)
text: (3) where γ is the weight of the augmented samples.

heading: 4 Experiments

heading: 4.1 Datasets
text: To evaluate the effectiveness of our proposed framework, we conduct experiments on two benchmarks of conversation summarization: SAMSum (Gliwa et al., 2019) and DialogSum (Chen et al., 2021c) which contain open-domain daily-chat conversations and diverse task-oriented conversations for real-life scenario. More detailed data statistics could be found in Table 7 in the Appendix.

heading: 4.2 Evaluation Metrics and Baselines
text: Evaluation Metrics We use the standard ROUGE metric1 (Lin, 2004) as automatic evaluation metrics, including ROUGE-1, ROUGE-2, and ROUGE-L for both SAMSum and DialogSum datasets. Note that the ROUGE scores might vary with different tookits.
Baselines with different augmentation strategy To demonstrate the superiority of our proposed compositional augmentation over previous data augmentation methods, we take several state-of-theart and representative data augmentation methods as baseline models. Specifically, they are tailored or suitable for conversation augmentation in different granularity including token-level, sentencelevel and context-level:
• BART (Lewis et al., 2020) is the state-of-theart pre-trained model for summarization. It also indicates training without augmentation. We use BART-base as well as BART-large as our base models for scalability.
1https://huggingface.co/spaces/ evaluate-metric/rouge
• Synonym Replacement (SR) (Kumar et al., 2020; Kobayashi, 2018) is a token-level approach, which keeps the semantic meaning unaffected by replacing a random word in the conversation with its synonyms.
• Back Translation (BT) (Chen et al., 2020a; Xie et al., 2019) is a utterance-level method, which firstly translates an selected utterance into an intermediate language, and then translates it back to the original language.
• Utterance Swapping (US) (Wang et al., 2021) is a context-level manner, which perturbs discourse relations to create augmented conversations. It first randomly selects two utterances in the conversation, and then swaps them.
• Semi-CODA (Chen and Yang, 2021a) is a two-stage noisy self-supervised framework that synthesizes a set of augmentation techniques, including random swapping and deletion, dialogue-acts-guided insertion, and conditional-generation-based substitution.

heading: 4.3 Implementation Details
text: During the training process, the encoder and decoder share the same set of parameters, which are initialized using a pre-trained BART (Lewis et al., 2020). The maximum iteration for self-training K is set to 5. During training, we used a batch size of 16 for 10 iterations with a 3e-5 learning rate. To ensure the model receives the same amount of
data for each training epoch, we replicate the original dataset to the same size as the augmentation datasets in the training stage. It takes around 5 hours to train on 4 A6400 GPUs for a full dataset under self-training, and 1 hour for the limited data setting. For joint training, it takes around 20 minutes for limited data, and 2 hours for full data. Note that the total amount for training (2x of the original samples) is equal for both self-training and joint training. Therefore, it is fair to directly compare those results. We take the average of 5 runs on random seeds for the main results shown in Table 1 and Table 2.

heading: 4.4 Results
text: Table 1 and Table 2 show the results on SAMSum and DialogSum2 benchmark datasets under both limited-data and full-data settings. Based on the numbers, we have the following observations: Different amount of data: When all the labeled data are used for training, COMPO shows performance gains compared to all the baseline methods, suggesting our method’s effectiveness as it works well even when a large number of data are used in the training process. With the limited data setting, we can see that performance gains are even larger compared with the full data setting. When less labeled data (i.e., 1% of the total data) are incorporated into the training process, the performance increase proves to be larger. Specifically, COMPO achieved an increase of 7.2% on Rouge-1, 14.0%
2Since there are three reference summaries on DialogSum test set, the results here are the average of three scores.
on Rouge-2, and 7.8% on Rouge-L compared with BART-base when 1% of the labeled data is used. Different backbone models: We also test COMPO’s scalability using both the BARTbase and BARTlarge as backbone pre-training models. Performance increases for both two PLMs on two datasets. With BARTbase, our method even outperforms BARTlarge baseline on SAMSum. With BARTlarge, COMPO also achieves consistent performance gains, which means COMPO is scalable to different backbone models. Not surprisingly, the increase is much larger with BARTbase. Different training settings: COMPO improves the performance of summarization under both selftraining and joint-training settings. While selftraining (leverage teacher model to predict pseudo summaries and trained for more iterations) surpasses joint-training, we can see that our newly generated summary labels are feasible to improve the performance over baseline models. Different datasets: Our model also performs well on DialogSum, which is a more abstractive, opendomain, and spoken analogous (Chen et al., 2021c) summarization dataset. We can infer that COMPO has great summarization ability when it comes to more challenging tasks.

heading: 4.5 Human Evaluation
text: We conducted human evaluations to assess the summaries generated by different models trained on 1% (147) conversations from the SAMSum dataset and 1% (125) conversations from the DialogSum dataset. Specifically, we asked annotators from
Amazon Mechanical Turk3 to rank summaries on a scale of 1 (the least preferred) to 3 (the most preferred). Summaries to be ranked are generated from BARTbase, COMPObase in self-training (COMPOsf) and joint-training (COMPO-jt) respectively. To avoid bias, we randomly sample summaries generated from 100 conversations for each dataset and perturb them for the workers to rank. Workers were paid 0.1$ for each ranking task. Every summary was ranked by three workers, and the rank for every summary was aggregated by majority voting. The Intra-Class Correlation (Koo and Li, 2016) (ICC1k) was 0.573, indicating moderate agreement. As shown in Figure 3, COMPO-sf and COMPO-jt both surpass the BART-base by a large margin on SAMSum and DialogSum datasets. Additionally, we observe larger gaps in terms of the scores for three models on DialogSum dataset. More details for human evaluation including interface design, scheduling details, and how we process with obtained rank scores could be found in Appendix D. Case studies for these three models could be found in Appendix E, where we provide the original conversation and the ranked three summaries.

heading: 5 Analysis

heading: 5.1 Automatic Quality Analysis of Summaries
text: We adopt a multi-dimensional evaluator (Zhong et al., 2022b) to evaluate the quality of our summaries automatically, in terms of coherence (coh.), consistency (con.), fluency (flu.), and relevance (rel.). Summaries generated with BARTbase,
3https://www.mturk.com/
COMPO-sf, and COMPO-jt are taken for comparison. As shown in Table 3, both COMPO-jt and COMPO-sf achieve better results against the baseline model, with 8% and 14% improvement on overall scores respectively. We also observe the largest performance increase on relevance. This indicates that summaries generated with COMPO are more factually consistent with conversations and accurately reflect important information.

heading: 5.2 Transferibility to other Datasets
text: To test whether COMPO is transferable to other input forms and datasets, we conduct experiments on CNN/Dailymail (Hermann et al., 2015), a traditional text summarization dataset from the news report. We treat sentences in articles as utterances in conversations and conduct exactly the same operations for augmentation. Table 4 shows the result on CNN/Dailymail in the limited data setting with only 1% (2871) data used. Consistent performance is achieved with evaluations in Section 4.4, our introduced COMPO significantly outperforms the baseline models. This verifies an additional generalization ability of our augmentation framework as well as the newly generated labels.

heading: 5.3 Ablation Studies
text: To see the effect of different components in COMPO, we conduct ablation studies on SAMSum dataset under the limited data setting, where 1%
labeled data are used for training.
Number of iterations K in self-training We explored how performance changes with the progress of self-training. Specifically, we use the number of iterations to identify. As shown in Table 6, the performance continues to increase until iteration 3, and then starts to fall. This suggests that the model could indeed learn from the teacher model as it generates the pseudo summaries as labels.
Effect of different components We tested the performance of using the traditional OpenIE method for action extraction. As shown in Table 5, COMPO, which leverages more diverse patterns for action extraction and syntactic structure, outperforms OpenIE. More examples of action extraction are listed in Appendix B.
We also conduct experiments with respect to alternative choices of action. Firstly, representations for conversation snippets are directly used for selective retrieval instead of extracted actions. Results show that using conversation snippets underperforms much, and even demonstrates the similar performance of BT. The potential reason is that directly using conversation snippets may bring some noise such as stopwords, pronouns, etc., instead of focusing on the core idea of a conversation snippet. We also try other structures such as Semantic Role Labeling (SRL) (Carreras and Màrquez, 2005), which is known to extract the predicate, theme, and recipient. As shown in Table 5, overall performance is not comparable to actions. We interpret this result from the following aspects: (i) num of SRL (avg 29.80) is far more than actions (avg 12.32) since SRL contains many prevalent but
noisy verbs such as “am”. (ii) average length of the extracted span is very long (sometimes even containing clauses) for SRL (avg 8.37) compared with actions (avg 4.74). Finally, we show the effect of selective retrieval against K-NN search. Unsurprisingly, K-NN search fails to outperform selective retrieval. This is because selective retrieval brings more coverage and diversity.
Augmentation with DialoGPT To investigate how COMPO surpasses model pre-trained on rough data as DA techniques, we experiment with DialoGPT (Zhang et al., 2020b). It is pre-trained on Reddit comment chains, which is easy to collect compared with human-labeled data. We follow the settings in (Feng et al., 2021b) and apply DialoGPT to generate the responses for each selected utterance. Then we treat them as newly augmented data samples for further training. As shown in Table 5, employing DialoGPT underperforms COMPO. The reasons are two folds: (i) DialoGPT fails to consider the structural and compositional information in the conversations, but rather generates plain responses. (ii) DialoGPT is pre-trained without speaker information, and thus may not be sensitive enough to tell the specific actions that happened.

heading: 6 Conclusion
text: This paper introduced a simple and effective compositional data augmentation method for abstractive conversation summarization. We leverage the topical view of conversations and treat them as the units for compositional operation. Extensive experiments on benchmark datasets demonstrate that COMPO significantly outperforms prior state-ofthe-art baselines in terms of both quantitative and qualitative evaluation, through generating compositional and diverse augmented data. Our method has key implications for designing augmentation techniques for low-resource dialogue-related tasks.
Limitations
Our work on COMPO is subject to multiple limitations. The first limitation is around its scope when probing compositional operations. We only explored compositional substitution for topical snippets in conversations as an initial effort. However, there are many other types of conversation structures that can be leveraged such as conversation stages or specific discourse acts. Second, we used a set of external tools to process the conversations for augmentation, such as the use of C99 for topic split and action extraction. Although we choose to select widely-used tools with high precision, error cascades are inevitable. Furthermore, our approach may not be applicable to low-resourced languages since these pre-processing tools may not be available even in the first place for these low-resourced contexts. We urge future work to further work on this line of compositional data augmentation without any dependencies on external software.
Ethics Statement
Despite the recent success of pre-trained language models in abstractive conversation summarization, they mostly rely on large-scale annotated data. This leads to a major concern about the labor-intensive and time-consuming annotating process, which might not be available for small research groups or institutions with relatively fewer resources; we hope that COMPO can be an initial effort in mitigating this issue. Our work also sheds light on a more general framework to deal with data scarcity issues, making summarization systems more applicable to real-world scenarios where annotations are often hard to get. Overall, we do not foresee any major risk or negative societal impact of our work. However, like any other machine learning model, the proposed framework may not be completely accurate and should be used with caution in real-world applications. To encourage reproducibility, we provide our source code in the supplementary material. The details of our framework are described in Section 3. The hyperparameters for our model are discussed in Section 4.1 and Section 4.3. The SAMSum and DialogSum datasets we experiment with are also publicly available resources.

heading: Acknowledgements
text: We thank members of the SALT Lab, and reviewers for their helpful feedback.

heading: A Statistics for Datasets
text: Here we provide the detailed statistics about the two datasets, SAMSum and DialogSum.
SAMSum contains open-domain daily-chat conversations in English written by linguists, each of which is annotated with summary by language experts. The topics contain arranging meetings, planning travels, chit-chat and so on. There are 14,732 dialogue-summary pairs for training, 818 and 819 instances for validation and test, respectively.
DialogSum is a large-scale dataset for real-life scenario conversations. It contains diverse task-oriented conversations. Specifically, speakers in DialogSum are denoted with #Person_1# and #Person_2#. The public dataset consists of 12,460 training samples. The validation and test set have equal 500 instances.
As could be inferred from Table 7, the number of participants for DialogSum are mostly 2, while SAMSum could have multi-party conversations. Also, the number of turns and reference length in DialogSum is shorter, which means that the information flow in DialogSum are relatively compact.

heading: B Details for Human Evaluation on Amazon MTurk
text: The web interface for human evaluation of quality is shown in Figure 4. Given a conversation, we randomly perturb the summaries generated and ask the workers to rank the summaries through the sliders. In principle, we do not accept repeated scores for three summaries since this is a ranking task. However, in practice, we found that there are almost identical summaries and it is difficult for human annotators to distinguish them. Therefore, for those cases (17 samples for SAMSum and 21 samples for DialogSum), we allow repetitive scores. For example, if all three summaries are identical, we will rank them as “1,1,1”. If two of the summaries are identical, we will rank them as “1,2,2” or “1,1,2”.

heading: C Patterns and Examples for Action Extraction
text: For action extraction, we first use a dependency parser to get the parsing tree, and we select all nonauxiliary verbs as centric tokens. Then we match the syntactic relations between the verbs and other spans/tokens to see if they match the predefined patterns. As shown in Table 9, there are some typical patterns used in the extraction, and their corresponding examples. For example, for pattern n1-nsubj-v1xcomp-a/n2, ’nsubj’ is the active relation between a noun and a verb. ’xcomp’ here indicates open clausal complement or predicative complement.

heading: D Examples for Actions Retrieval
text: In this section, we display different actions retrieved with selective retrieval and traditional kNN method to provide an intuitive view of their effects, and how they influence the final performance of summarization. For each of the actions, top three retrieved samples are listed for both the selected retrieval and kNN method. As can be seen, traditional kNN method usually focus only on word semantics, and is not able to generate diverse results.

heading: E Examples for Summaries generated from three models mentioned in Section 4.5
text: We demonstrate several cases for summary generation with BART-base, COMPO-jt, COMPO-sf. We also attach groundtruth summaries for reference in Table 10. For each summary generated, the human evaluation scores (after majority voting) are also provided.

heading: F Examples for Newly Augmented Data
text: In this section, we provide several examples for the newly augmented data generated with COMPO, as shown in Table 11. Selected topical split for compositioanl operation is highlighted in green.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Section 7

heading: 3 A2. Did you discuss any potential risks of your work?
text: Section 8

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Section 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Section 3.2

heading: 3 B1. Did you cite the creators of artifacts you used?
text: Section 3.2, Section 4
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? No response.
7 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? The original intended use is not found
7 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Left blank.
7 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Left blank.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Appendix A, Section 4
C 3 Did you run computational experiments? Section 4

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Section 4.3
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Section 4.3

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 4.3

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section 4.3
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? Section 4.5, Appendix b

heading: 3 D1. Did you report the full text of instructions given to participants, including e.g., screenshots,
text: disclaimers of any risks to participants or annotators, etc.? Appendix B

heading: 3 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students)
text: and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Section 4.5

heading: 3 D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to
text: crowdworkers explain how the data would be used? Section 4.5
7 D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Left blank.
7 D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Left blank.

heading: Future Work
text: We urge future work to further work on this line of compositional data augmentation without any dependencies on external software