Evaluating Robustness of Open Dialogue Summarization Models in the Presence of Naturally Occurring Variations (2024)

Abstract: Dialogue summarization involves summariz001 ing long conversations while preserving the 002 most salient information. Real-life dialogues 003 often involve naturally occurring variations 004 (e.g., repetitions, hesitations), and in this study, 005 we systematically investigate the impact of 006 such variations on state-of-the-art open dia007 logue summarization models whose details are 008 publicly known (e.g., architectures, weights, 009 and training corpora). To simulate real-life 010 variations, we introduce two types of pertur011 bations: utterance-level perturbations that mod012 ify individual utterances with errors and lan013 guage variations, and dialogue-level perturba014 tions that add non-informative exchanges (e.g., 015 repetitions, greetings). We perform our analy016 sis along three dimensions of robustness: con017 sistency, saliency, and faithfulness, which aim 018 to capture different aspects of performance of 019 a summarization model. We find that both fine020 tuned and instruction-tuned models are affected 021 by input variations, with the latter being more 022 susceptible, particularly to dialogue-level per023 turbations. We also validate our findings via hu024 man evaluation. Finally, we investigate whether 025 the robustness of fine-tuned models can be im026 proved by training them with a fraction of per027 turbed data and find that this approach does not 028 yield consistent performance gains, warranting 029 further research. Overall, our work highlights 030 robustness challenges in current open models 031 and provides insights for future research. 032

text: Proceedings of the 6th Workshop on NLP for Conversational AI (NLP4ConvAI 2024), pages 56–72 August 16, 2024 ©2024 Association for Computational Linguistics
Dialogue summarization involves summariz-001 ing long conversations while preserving the002 most salient information. Real-life dialogues003 often involve naturally occurring variations004 (e.g., repetitions, hesitations), and in this study,005 we systematically investigate the impact of006 such variations on state-of-the-art open dia-007 logue summarization models whose details are008 publicly known (e.g., architectures, weights,009 and training corpora). To simulate real-life010 variations, we introduce two types of pertur-011 bations: utterance-level perturbations that mod-012 ify individual utterances with errors and lan-013 guage variations, and dialogue-level perturba-014 tions that add non-informative exchanges (e.g.,015 repetitions, greetings). We perform our analy-016 sis along three dimensions of robustness: con-017 sistency, saliency, and faithfulness, which aim018 to capture different aspects of performance of019 a summarization model. We find that both fine-020 tuned and instruction-tuned models are affected021 by input variations, with the latter being more022 susceptible, particularly to dialogue-level per-023 turbations. We also validate our findings via hu-024 man evaluation. Finally, we investigate whether025 the robustness of fine-tuned models can be im-026 proved by training them with a fraction of per-027 turbed data and find that this approach does not028 yield consistent performance gains, warranting029 further research. Overall, our work highlights030 robustness challenges in current open models031 and provides insights for future research.032
1 Introduction033
Real-life conversations often exhibit a wide range034 of language variations, including typographical er-035 rors, grammatical mistakes, and certain exchanges036 such as repetitions and speaker interruptions, which037 are unrelated to the primary purpose of the con-038 versation (Sacks et al., 1974). However, existing039 dialogue summarization datasets, which are used040 to train current summarization models, do not ad-041 equately capture these variations, as they are typi-042 cally constructed by annotators simulating specific043
scenarios (Yuan and Yu, 2019) or extracted from 044 English-speaking practice websites (Gliwa et al., 045 2019). Even the datasets consisting of real-life 046 conversations (Feigenblat et al., 2021) can exhibit 047 only a limited range of variations owing to practi- 048 cal limitations posed by the data collection process 049 (e.g., high or low prevalence of conversations from 050 different social demographics). Consequently, dia- 051 logue summarization models deployed in business 052 scenarios encounter diverse variations not observed 053 during training. This raises a crucial question: Can 054 current dialogue summarization models effectively 055 handle conversations with naturally occurring vari- 056 ations that are legitimate inputs but not observed in 057 the training data? 058
In this work, we study the impact of naturally 059 occurring variations on the performance of the 060 state-of-the-art open dialogue summarization mod- 061 els (with publicly known architecture, weights, 062 and training corpus) using three publicly avail- 063
1 56
able datasets. We examine the performance of064 encoder-decoder Transformer models in two setups065 a) fine-tuned on specific dialogue summarization066 datasets (Lewis et al., 2020; Zhang et al., 2019;067 Raffel et al., 2020b), and b) instruction-tuned mod-068 els which have shown impressive zero-shot perfor-069 mance more recently (Gupta et al., 2022; Chung070 et al., 2022). Such models are often preferred071 in high-stakes business settings (e.g., medical, le-072 gal, and customer support) over proprietary models073 (e.g., ChatGPT), owing to user privacy concerns.074
To simulate variations we design two kinds of075 perturbations: (a) utterance-level perturbations,076 and (b) dialogue-level perturbations (defined in077 Section 3), which are inspired by common real-life078 interaction patterns from the Natural Conversation079 Framework (Moore and Arar, 2019). We evaluate080 the performance of summarization models along081 three conceptually distinct robustness dimensions—082 consistency, saliency, and faithfulness—and elabo-083 rate on their empirical relationship.084
Our analysis reveals that both fine-tuned and085 instruction-tuned models are impacted by utterance086 and dialogue-level perturbations. Instruction-tuned087 models are impacted more than fine-tuned models088 and are also more susceptible to dialogue-level per-089 turbations than utterance-level perturbations. Both090 types of models show a preference for information091 from repeated, long, and leading utterances in the092 dialogue. Figure 1 shows an example where the093 model includes repeated utterances in the summary,094 whereas the non-repeated original utterance wasn’t095 included in the summary before perturbation. We096 also validate our findings via human evaluation.097
Finally, we investigate whether fine-tuned mod-098 els improve by training with perturbed data. We099 find that this approach does not consistently en-100 hance performance, and different perturbations re-101 quire varying amounts of training examples for102 gains. Thus, further research is needed to address103 these robustness challenges.104
2 Related Work105
Prior work has investigated the robustness of lan-106 guage understanding models mainly focusing on107 classification tasks (Moradi and Samwald, 2021).108 Some dialogue-related classification tasks have109 also been explored, including dialogue act predic-110 tion (Liu et al., 2021), intent detection and slot111 tagging (Einolghozati et al., 2019; Sengupta et al.,112 2021), state tracking and dialogue modeling (Cho113 et al., 2022; Tian et al., 2021; Zhu et al., 2020; Kim114 et al., 2021; Peng et al., 2020).115
Some studies have also investigated the robust- 116 ness of neural language generation models, includ- 117 ing neural machine translation (Niu et al., 2020; 118 Karpukhin et al., 2019; Vaibhav et al., 2019), ques- 119 tion answering (Peskov et al., 2019), and open do- 120 main multi-document summarization (Giorgi et al., 121 2022). However, some of these studies consider 122 perturbations that are of extreme nature (e.g., ran- 123 dom shuffling and deletion of words) and may oc- 124 cur rarely in the real world. Ganhotra et al. (2020) 125 investigated the impact of natural variations on re- 126 sponse prediction tasks in goal-oriented dialogues. 127
For summarization task in particular, previous 128 studies focused on summarizing news articles and 129 documents (Jing et al., 2003; Meechan-Maddon, 130 2019; Krishna et al., 2022). However, the nature 131 of noise in a multi-party dialogue differs signifi- 132 cantly from noise in documents. While some types 133 of noise (e.g., spelling mistakes, grammatical er- 134 rors) could occur in both, the patterns such as rep- 135 etitions, reconfirmations, hesitations, and speaker 136 interruptions (Sacks et al., 1974; Feng et al., 2021; 137 Chen and Yang, 2021) are peculiar to dialogues, 138 posing unique challenges for accurate and robust 139 summarization. The focus of this work is to assess 140 the robustness of dialogue summarization models 141 in the presence of naturally occurring variations, 142 which has been understudied in the prior literature. 143
3 Simulating Naturally Occurring 144 Variations 145
To introduce naturally occurring variations in con- 146 versations, we consider two kinds of simulated 147 perturbations, utterance-level and dialogue-level. 148 We apply each perturbation individually to a dia- 149 logue to study its impact systematically. Our per- 150 turbations are inspired by the Natural Conversation 151 Framework (Moore and Arar, 2019), created after 152 analyzing real-world conversations across various 153 use cases and provides common interactive pat- 154 terns that occur in real life.1 Appendix A.1 lists 155 examples for each perturbation. 156
3.1 Utterance-level Perturbations 157 The utterance-level perturbations modify a single 158 utterance and are adapted from (Liu et al., 2021). 159 We perturb each utterance of the dialogue. For per- 160 turbations where multiple words in an utterance 161 can be perturbed (e.g., spelling mistake, character 162 casing), we consider only low-modification levels 163 (i.e., perturb a word with 0.2 probability), which 164
1Some examples include patterns such as C1.0 (opening greeting agent), C4.6 (closing success check), B2.1.0 (repeat request), A2.8 (hold request).
2
also cause a considerable change in model perfor-165 mance.2166
Typographical Errors Typographical errors oc-167 cur when participants try to type quickly in chat-168 based interactions. We use simple regex-based per-169 turbations, e.g., punctuation marks removal, whites-170 pace removal or addition, changing letter casing,171 and substitutions of common expansions and con-172 tractions. We introduce spelling errors following173 the approach of Yorke as used in (Mille et al., 2021),174 replacing random letters with other letters closely175 co-located on the keyboard positions. We ensure176 that mistakes are not introduced in a proper-noun177 phrase (e.g., restaurant name) to avoid changes in178 important information.179
Grammatical Errors We focus on two frequent180 grammatical errors: dropping determiners and181 subject-verb disagreements. To drop determiners,182 we drop all the words in a sentence with the DET183 tag. To introduce subject-verb disagreement, we184 identify auxiliary verbs (via AUX tag) and convert185 between plural and singular forms as appropriate,186 keeping the tense unchanged.187
Language-use Variations Users can vary in their188 choices of dialect and vocabulary. We consider189 three language-use perturbations: substituting ad-190 jectives with synonyms, inflectional variations, and191 synthetic African American Vernacular English192 (AAVE) dialect. For synonym substitution, we sub-193 stitute adjectives in an utterance with their WordNet194 (Miller, 1998) synonyms. To introduce inflectional195 variations, we follow the approach proposed in196 Dhole et al. (2021), where we lemmatize each con-197 tent word in an utterance, randomly sample a valid198 POS category, and re-inflect the word according to199 the chosen category. To transform an utterance to200 synthetic AAVE dialect, we use the set of lexical201 and morphosyntactic transformation rules proposed202 by Ziems et al. (2022).203
3.2 Dialogue-level Perturbations204
We introduce new utterances that contribute no205 additional information, to test a model’s ability to206 focus on the overall meaning of a conversation and207 identify salient information.208
Repetitions Repeating and rephrasing occur209 commonly in real-life spoken conversations. In210 this perturbation, we randomly select an utterance211
2See Appendix A.5 for analysis with different perturbation rates.
to repeat.3 We then inject a synthetic utterance 212 requesting the other participant to repeat the in- 213 formation (e.g., ‘Sorry, I couldn’t hear you, can 214 you repeat?’).4 Since humans tend to rephrase the 215 original message slightly instead of repeating it 216 verbatim, we paraphrase the original utterance be- 217 fore including it as a response to the request for 218 repetition. We use Qian et al. (2019)’s paraphraser 219 for this task. The rest of the dialogue remains un- 220 changed. This perturbation allows us to examine 221 repetition bias; i.e., does the model consider re- 222 peated utterances more significant, even when they 223 do not contain important information? 224
Time delays A participant may ask the other 225 party to wait while they gather information. To 226 simulate this, we add three synthetic utterances 227 consecutively: a request to wait (e.g., ‘Just give 228 me a few minutes.’), an acknowledgment from the 229 other participant (e.g., ‘Sure’), and an expression 230 of gratitude from the first participant (e.g., ‘Thanks 231 for waiting.’). These utterances are inserted after 232 a randomly selected utterance from the participant 233 being asked to wait. 234
Greeting and closing remarks It is also com- 235 mon to begin a conversation with a friendly greet- 236 ing and end with some closing remarks. For the 237 greetings perturbation, we insert a greeting as the 238 first utterance, such as ‘Hi! I am your customer 239 support assistant. How may I help you today?’ in 240 customer support dialogues and ‘Hey there!’ in 241 open-domain chit-chat. For the closing remarks 242 perturbation, we insert a final message: ‘Thank 243 you for contacting us.’ in customer support dia- 244 logues and ‘Cool, talk to you later!’ in open domain 245 chit-chat. Each perturbation is applied individually 246 to a dialogue. Both of these perturbations help 247 us investigate structural biases present in dialogue 248 summarization models, also known to impact news 249 summarization models (Xing et al., 2021; Jung 250 et al., 2019). For instance, the greeting perturba- 251 tion helps examine lead bias (preference for the 252 first utterance), and closing remarks perturbation 253 helps examine recency bias (preference for the last 254 utterance). 255
3See Appendix A.4 for targeted perturbations, where we select an utterance to repeat based on its saliency.
4We use this utterance to operationalize the repetition perturbation, inspired by spoken dialogues. However, repetitions can also appear in written dialogues (e.g., sending the same message multiple times to ensure communication, emphasizing points, or dealing with technical issues.). Furthermore, models trained on written dialogues are often deployed to summarize transcripts of spoken dialogues, where such utterances are more common.
3
Split and combined utterances In chat-based256 conversations, participants can have varying prefer-257 ences for either conveying information over multi-258 ple consecutive utterances or sending one long mes-259 sage. To simulate split utterance perturbation, we260 divide a randomly sampled utterance into consec-261 utive utterances by splitting it at every five words.262 Conversely, to simulate combined utterance per-263 turbation, we identify sequences of consecutive264 utterances from a single participant in a dialogue265 and concatenate them. We combine consecutive ut-266 terances from only one participant at a time. Each267 perturbation is applied individually to a dialogue.268 Both these perturbations allow us to examine long269 bias (the model’s preference to include a long utter-270 ance over shorter utterances, even when multiple271 short utterances include salient information).272
3.3 Quality evaluation of perturbed dialogues273 We conduct a human validation of the perturbed274 dialogues. The goal of this evaluation is to ensure275 that our perturbations do not alter the dialogue’s276 meaning or introduce new information, thereby277 validating the quality of our perturbed test set. We278 sample 20 dialogues and their summaries from each279 of the three datasets (§5.1) and perturb each dia-280 logue with all of the utterance and dialogue-level281 perturbations, resulting in a total of 480 dialogues.282 Two annotators are asked to determine whether283 the reference summary for the original dialogue re-284 mains valid for all the perturbed dialogues (see Ap-285 pendix A.2 for details on annotation guidelines). In286 cases of disagreement, a third annotator breaks the287 tie. The annotators marked 97.5% of the perturbed288 dialogues as being reasonably summarized by the289 summary of the original dialogue, thus validating290 the use of proposed perturbations to investigate the291 robustness of dialogue summarization models. Our292 human evaluation also suggests that our perturba-293 tions do not drastically alter the dialogue and the294 dialogues remain readable and semantically con-295 sistent. Otherwise, for an altered dialogue, the296 original summary would have been marked invalid.297
4 Quantifying Robustness298
For tasks involving text generation, such as di-299 alogue summarization, measuring robustness in-300 volves determining the relationship between differ-301 ent pairs of natural language texts. As a result, the302 robustness of generative tasks is less well-defined,303 compared to a classification task (Liu et al., 2021)304 and can manifest in several ways. We consider305 three dimensions for measuring robustness issues306 that can arise in dialogue summarization.307
Let x denote the original dialogue, yr be the 308 reference summary of the original dialogue, f be 309 the summarization model trained on (x, yr) ∼ D, 310 and f(x) be its prediction over x. Let x′ = x + 311 δ denote the perturbed dialogue and f(x′) be its 312 predicted summary. 313
Consistency A model is consistent (and hence 314 robust) under a perturbation (δ) if the two sum- 315 maries, f(x) and f(x′ = x+ δ), are semantically 316 similar, resulting in minimal change. We quantify 317 the change in model-generated output as follows, 318
∆zc = |SCORE(f(x), f(x))− SCORE(f(x), f(x′))|
SCORE(f(x), f(x)) (1) 319
further simplified as, 320
∆zc = 1− SCORE(f(x), f(x′)) (2) 321
where SCORE is any text similarity metric (e.g., 322 BERTScore) that assigns a value of 1 for identical 323 inputs and 0 for dissimilar inputs. By definition, 324 ∆zc ∈ [0, 1]. Note that consistency is sufficient but 325 not necessary for robustness: a good summary can 326 be expressed in diverse ways, which leads to high 327 robustness but low consistency. 328
Saliency Assuming that the reference summary 329 includes the most salient information conveyed 330 in the input dialogue, we compute the change 331 in salient information captured by the model- 332 generated summaries (before and after perturba- 333 tion) w.r.t the reference summary as follows: 334
∆zs = |SCORE(yr, f(x))− SCORE(yr, f(x′))|
SCORE(yr, f(x)) (3) 335
where SCORE is any text similarity metric (e.g., 336 BERTScore). Since ∆zs measures the normalized 337 change in similarity scores, ∆zs ∈ [0, 1]. 338 Faithfulness Faithfulness refers to the extent 339 to which the generated summary is supported by 340 the content of the input dialogue, thus accurately 341 reflecting the information without introducing spu- 342 rious or fabricated details, commonly termed as 343 hallucinations. We compute the change in faithful- 344 ness as follows: 345
∆zf = |SCORE(x, f(x))− SCORE(x, f(x′))|
SCORE(x, f(x)) (4) 346
where SCORE is any text-based precision met- 347 ric measuring the fraction of information in the 348 summary (f(x)) supported by the input dialogue 349
4
(x) (e.g., BERTScore-Precision). Since ∆zf mea-350 sures the normalized change in precision scores,351 ∆zf ∈ [0, 1]. Note that, the second term in the352 numerator compares x with f(x′) since we are353 interested in measuring the fraction of summary354 information supported by the ‘original dialogue.’355 Furthermore, since our added perturbations do not356 add any new information to the dialogue, x and357 x′ would essentially contain the same information.358 Clearly, for all three dimensions, the higher the ∆z,359 the lower the robustness of the model.360
5 Evaluating Robustness361
We present our key observations on how various362 perturbations impact the model performance.363
5.1 Implementation Details364 Datasets We consider two task-oriented dia-365 logues, TWEETSUMM (Feigenblat et al., 2021) and366 TODSum (Zhao et al., 2021), both consisting of367 conversations between an agent and a customer.368 TODSum comprises dialogues from multiple sub-369 domains (restaurants, movies, etc), collected via370 crowdsourcing where annotators are tasked to gen-371 erate dialogues based on a given scenario. In con-372 trast, TWEETSUMM focuses solely on customer sup-373 port conversations occurred at Twitter. We also374 include SAMSUM (Gliwa et al., 2019), a corpus of375 chit-chat dialogues between two or more friends.376
Models We analyze the robustness of three Trans-377 former based encoder-decoder models for dialogue378 summarization, Pegasus-large (568M parame-379 ters) (Zhang et al., 2019), BART-large (400M380 parameters) (Lewis et al., 2020) and T5-base381 (220M parameters) (Raffel et al., 2020a), whose382 details are publicly available. All models have383 a comparable number of parameters. We fine-384 tune each model on the train split of the respec-385 tive dataset. We use beam search5 with size 5 to386 generate summaries. We also investigate the robust-387 ness of instruction-tuned versions of two of these388 models, DIAL-BART0 (406M parameters) (Gupta389 et al., 2022) and FLAN-T5-large (783M parame-390 ters) (Chung et al., 2022), used as zero-shot sum-391 marizers, without fine-tuning on the three dialogue392 summarization datasets considered in this work.393
Metrics We evaluate summaries using394 BERTScore (Zhang et al., 2020), which has395 been shown to better correlate with human judg-396 ment (Fischer et al., 2022). BERTScore calculates397 precision, recall, and F1 scores by comparing a398
5Nucleus sampling omitted to avoid sampling variance.
model-generated summary to a reference summary. 399 We use F1 to compute consistency and saliency, 400 and precision to compute faithfulness. To 401 validate observed trends, we additionally evaluate 402 summaries using ROUGE-L metric (Lin, 2004), 403 which measures lexical overlap, and SummaC 404 metric (Laban et al., 2022), which measures 405 factual consistency. For all the reported results, we 406 observe similar trends via ROUGE-L and SummaC 407 (Tables 11,12,13 in Appendix A.8). While we 408 report results using these metrics, the three 409 robustness dimensions can be computed using 410 any evaluation metric. For each reported result, 411 we use a non-parametric bootstrap (Wasserman, 412 2004, ch. 8) to infer confidence intervals (CIs). We 413 utilize 104 bootstrap samples of the dialogues to 414 report 95% bootstrap CIs via the normal interval 415 method (Wasserman, 2004, ch. 8.3). 416
5.2 How robust are fine-tuned models? 417 Fine-tuned dialogue summarization models are 418 affected by both utterance and dialogue level 419 perturbations Table 1 shows the change in con- 420 sistency, saliency, and faithfulness owing to utter- 421 ance and dialogue level perturbations on all three 422 datasets. All three models are equally impacted by 423 various perturbations. Models trained on TweetSum 424 and SAMSum are impacted equally by both utterance- 425 level and dialogue-level perturbations. TODSum 426 is the least impacted, since this dataset contains 427 template-based summaries where only entities from 428 the dialogue are required to be filled. We see a ma- 429 jor impact on faithfulness, with the highest impact 430 on the model trained on the TODSum dataset. 431
Impact of utterance perturbations Table 2 432 shows that these perturbations have a compara- 433 ble impact (shown averaged over all three models). 434 Models trained on TODSum exhibit little change in 435 consistency and saliency, but a significant change 436 in faithfulness. This is expected since the TODSum 437 summaries are extractive, following a pre-defined 438 template, and only require substituting entity infor- 439 mation extracted from the dialogue. Since the tem- 440 plate is fixed and the summaries can only change 441 in entity information before and after perturbation 442 and w.r.t reference summary, we see a small change 443 in consistency and saliency. However, we observe 444 a large change in faithfulness, as this dimension 445 focuses on the factual correctness of the summary. 446
Impact of dialogue perturbations: Table 3 re- 447 ports the impact of dialogue-level perturbations 448 (averaged over all models) and shows significant 449 changes for repetition, time delays, greetings, and 450
5
split utterances. For instance, when subjected to451 repetitions, the models tend to include repeated452 utterances in the summary, even if they were pre-453 viously deemed unimportant (repetition bias; Fig-454 ure 1). Additionally, the models demonstrate a455 preference for the first utterance in a dialogue (lead456 bias), rendering them susceptible to greetings per-457 turbation. This observation aligns with prior find-458 ings for news summarization, where sentences at459 the beginning of an article are more likely to con-460 tain summary-worthy information. Similarly, in461 customer-support conversations, the first utterance462 frequently addresses the primary issue faced by the463 customer. Consequently, models trained on such464 datasets exhibit lead bias. Finally, the models pre-465 fer lengthy utterances in the summary (long bias),466 by being more affected by split perturbations, and467 less affected by short utterances combined.468
5.3 Effect of model size on robustness469 Table 4 shows the change in consistency for models470 with different number of parameters: BART-base,471 BART-large, T5-base, and T5-small. The mod-472 els are almost equally affected by perturbations,473 irrespective of size, suggesting that robustness is-474 sues cannot be mitigated by scaling the model size.475
5.4 How robust are instruction-tuned models476 when used as zero-shot summarizers?477
DIAL-BART0 and FLAN-T5-large are instruction-478 tuned on multiple tasks, with DIAL-BART0, in479 particular, is instruction-tuned on dialog-specific480 tasks. However, neither model was trained on the481 TweetSum dataset, providing a zero-shot setting482
to evaluate their dialogue summarization capabil- 483 ities. As depicted in Table 5, both DIAL-BART0 484 (∆zc=30.37% for utterance and 34.30% for dia- 485 logue) and FLAN-T5 (∆zc=38.23% for utterance 486 and 44.12% for dialogue) are much more sensitive 487 to perturbations compared to their fine-tuned coun- 488 terparts (∆zc=17.36% for utterance and 16.82% 489 for dialogue, averaged over three models). 490
In contrast to fine-tuned models, the zero-shot 491 models are affected more by the dialogue-level 492 perturbations (∆zc=34.30% for DIAL-BART0 and 493 ∆zc=44.12% for FLAN-T5) than utterance-level 494 perturbations (∆zc=30.37% for DIAL-BART0 and 495 ∆zc=38.23% for FLAN-T5). Among utterance-level 496 perturbations, similar to the fine-tuned models, 497 zero-shot models are also impacted equally by all 498 perturbations. Among dialogue-level perturbations 499 as well, similar to the fine-tuned models, zero-shot 500 models are most impacted by repetitions, greetings 501 and split utterances (Appendix A.6). 502
We additionally consider a recent instruction- 503 tuned large language model, Llama-2-70B, with 504 only publicly available weights. This model is also 505 significantly larger (70B) than the other models 506 (<0.9B). Our results show high sensitivity to per- 507 turbations for this model (∆zc=47.10% for utter- 508 ance and ∆zc=54.53% for dialogue perturbations), 509 though we leave detailed human evaluation of the 510 outputs of this model for future work. 511
5.5 Validity of findings with human evaluation 512
We conduct another human evaluation to con- 513 firm the trends observed with automatic similar- 514 ity metrics. Specifically, we collect similarity 515 scores between summary pairs using human an- 516 notations instead of automated similarity metrics 517 (e.g., BERTScore). The goal is to ensure that ro- 518 bustness trends observed with automated metrics 519 are similar to those from human evaluation. 520
We use the consistency dimension for this eval- 521 uation for two main reasons: 1) Empirically, the 522 three robustness dimensions exhibit a strong cor- 523 relation (Table 10). Thus, using any of the three 524
6
dimensions would suffice for human evaluation,525 and (2) Among the three dimensions, consistency526 is easiest to use for human evaluation since it only527 requires the comparison of two summaries.528
We collected annotations via the Appen platform529 (https://appen.com/), asking annotators to com-530 pare summaries of the perturbed and unperturbed531 dialogue, ranking their similarity on a Likert scale532 of 1 (dissimilar) to 4 (identical or paraphrases). To533 collect annotations, we used the same set of 20 dia-534 logues as in §3.3 from the TweetSum dataset. Each535 dialogue was perturbed with one of the eight cate-536 gories (utterance- and dialogue-level), yielding 160537 summary pairs to be annotated.538
We collected 3 annotations per summary pair,539 totaling 480 annotations; after filtering out noisy540 annotations, we conducted our analysis on the re-541 maining 314 examples (Appendix A.3 provides an-542 notation procedure and guidelines). We aggregate543 annotations using majority voting to get similar-544 ity scores. To compute consistency scores (equa-545 tion 1), we map the Likert scale to continuous nu-546 meric scores from 0 to 1. We compute mean scores547 across all pairs for a given dataset and perturbation.548
As shown in Figure 2, we observe similar trends,549 with models exhibiting repetition, long, and lead550 biases, and that models are affected nearly equally551 by all utterance perturbations. While the absolute552 values of ∆zc differ between calculations using au-553 tomatic metrics and human annotations, the relative554 impact of different perturbations on the model is555 similar. For instance, combined utterances and clos-556 ing remarks have the least impact than repetition,557 greetings, and split utterance perturbations.6558
6Except time delays, owing to noise in human annotations.
5.6 Relationship among dimensions 559
While theoretically, three dimensions (§4) measure 560 different aspects of robustness, empirically they ex- 561 hibit a strong correlation of > 84% across datasets 562 and models (details in Table 10 in Appendix). 563
This observation can be conceptually explained 564 to some extent. For instance, high saliency implies 565 high consistency: if summaries before and after per- 566 turbation are similar to the reference summary, they 567 will be similar to each other, leading to low ∆zs 568 and thus low ∆zc. Similarly, high saliency implies 569 high faithfulness: if the model-generated summary 570 is similar to the reference summary, it will also be 571 factually consistent with the input dialogue, lead- 572 ing to low ∆zs and thus low ∆zf . However, if 573 ∆zs is large, the model could remain faithful un- 574 der perturbation (small ∆zf ): summaries can be 575 different from the reference summary yet consis- 576 tent with the input dialogue. Thus, conceptually, 577
7
the relation can be explained in only one direction,578 but empirically the dimensions are highly corre-579 lated. Nevertheless, our findings are insightful in580 their own right, suggesting that the high correlation581 among all dimensions could be valuable for future582 robustness studies. For instance, the consistency583 or faithfulness dimension can serve as reference-584 free measures of robustness. Consistency is also585 the easiest to use for human evaluation, as it only586 requires comparing two summaries.587
6 Improving Robustness588
One solution to address robustness issues could be589 to employ reverse heuristics to remove perturba-590 tions from dialogues. However, not all perturba-591 tions can be easily discovered and removed. For592 example, in repetition or time delay perturbations,593 the repeated utterance may include less information594 or be paraphrased compared to the original. While595 greetings and closing remarks might be simpler596 to remove, we include these perturbations as they597 offer a systematic approach to investigating model598 behavior, such as potential lead and recency biases.599 Another potential solution to address robustness600 issues can be to use recent large language models to601 pre-process dialogues by removing errors and rep-602 etitions. However, this approach suffers from two603 challenges: (1) During deployment, additional pre-604 processing could increase latency, and (2) language605 models may hallucinate content, posing the risk of606 introducing factual errors in the input dialogue.607 Finally, we examine if training with perturba-608 tions can help to mitigate robustness issues. We609 fine-tune BART on the training data augmented with610 perturbations and re-evaluate its performance. We611 create multiple training datasets, each modified by612 a specific kind of perturbation (typographical errors613 and language use variations for utterance level; rep-614 etitions, split utterances, and greetings for dialogue615 level), using TweetSum’s training split. These mod-616 ified datasets, with 5-50% of dialogues perturbed,617 are used to fine-tune BART, which we then test on a618 similarly altered TweetSum’s test split.7 We hypoth-619 esize that training with more perturbed dialogues620
7We experimented with training and evaluating a single model on data with all perturbations. However, since different perturbations can have varied impacts on model performance, we found perturbation-wise analysis more interpretable.
will initially improve performance until a threshold, 621 after which overfitting may reduce effectiveness. 622
Figure 3 shows the change in model consistency 623 when fine-tuned with perturbations. The lower the 624 change in consistency, the higher the model robust- 625 ness to the perturbations. One takeaway is that 626 different perturbations necessitate varying amounts 627 of perturbed examples in the training set to achieve 628 maximum performance gain. For example, typo- 629 graphical errors and language use variations yield 630 the largest drop in ∆zc when approximately 40% 631 and 45% of the dialogues are perturbed during train- 632 ing. In contrast, dialogue-level perturbations re- 633 quire significantly less perturbed data during train- 634 ing, with approximately 30% split-utterances, 15% 635 greetings, and only 5% repetitions being sufficient. 636 Overall, the results demonstrate that fine-tuning 637 with perturbed data does not yield consistent per- 638 formance improvements, warranting more detailed 639 exploration as part of future work. 640
7 Conclusion 641 We investigate the impact of naturally occurring 642 variations on state-of-the-art dialogue summariza- 643 tion models using three publicly available datasets. 644 To simulate variations, we introduce utterance-level 645 and dialogue-level perturbations. We conduct our 646 analysis using three dimensions of robustness: con- 647 sistency, saliency, and faithfulness, which capture 648 different aspects of the summarization model’s per- 649 formance. Our results show that both fine-tuned 650 and instruction-tuned models are affected by pertur- 651 bations, with instruction-tuned models being more 652 susceptible, particularly to dialogue-level perturba- 653 tions, spurring the need for future research. 654
8
8 Limitations655
We list some of the limitations of our study which656 researchers and practitioners would hopefully ben-657 efit from when interpreting our analysis. 1) Our658 analysis uses automatic metrics to measure seman-659 tic similarity. Established metrics such BERTScore660 are imperfect (Deutsch et al., 2022). However, they661 are widely used in the summarization literature,662 and also correlate with human judgements of sum-663 mary quality, and thus are useful for comparing664 system-level performance. To validate our findings,665 we also conduct a human evaluation to better under-666 stand trends observed due to various perturbations.667 The investigation of better-automated metrics for668 natural language generation is an active field of re-669 search, and we hope to integrate novel performance670 metrics in future work. (2) While our perturbations671 are motivated by real-life scenarios, they are still672 synthetic in nature. However, we take care wher-673 ever possible to avoid unrealistic changes to the di-674 alogues. (3) Our study limits to only open-sourced675 models and does not investigate the robustness of676 proprietary LLMs (e.g., ChatGPT), which may be677 more robust. We decided to limit our study to open-678 sourced models as it allows us to carefully control679 what is in the training data, which is not possible680 with proprietary LLMs and the possibility of data681 contamination also makes it hard to draw conclu-682 sions. (4) Our study mainly focuses on text-based683 dialogue summarization datasets and does not in-684 clude spoken conversations, which would bring in685 very different and diverse nuances of spoken con-686 versations compared to text-based conversations,687 and is currently out of the scope of this paper. (5)688 Our study proposes one possible method to mea-689 sure robustness, and we acknowledge that there690 can be many other viable ways to quantify robust-691 ness. However, quantifying the robustness of tasks692 involving text generation (e.g., summarization) is693 an active area of research (Wang et al., 2022) and694 we hope our work will spur further investigation695 as part of future work. (6) We did not investigate696 the robustness of models under both utterance and697 dialogue level perturbations occurring together in698 a single dialogue, as that would result in a large699 number of possible combinations to consider. We700 leave this for future work.701
9 Ethics Statement702
All annotators in our human evaluation were re-703 cruited via Appen platform and were presented704 with a consent form prior to the annotation. They705 were also informed that only satisfactory perfor-706
mance on the screening example will allow them 707 to take part in the annotation task. None of the ma- 708 terial/examples they looked at had any hateful or 709 abusive content. We also ensured that the annota- 710 tors were paid fair amount of wages using Appen’s 711 Fair Pay Price Per Judgment which equates to an 712 hourly rate matching a little over the minimum 713 wage of annotators in their respective countries. 714 All the datasets used in this work are publicly avail- 715 able under the CDLA-Sharing license and do not 716 contain any private information. 717
References 718 Jiaao Chen and Diyi Yang. 2021. Simple conversa- 719
tional data augmentation for semi-supervised abstrac- 720 tive dialogue summarization. In Proceedings of the 721 2021 Conference on Empirical Methods in Natural 722 Language Processing, pages 6605–6616, Online and 723 Punta Cana, Dominican Republic. Association for 724 Computational Linguistics. 725
Hyundong Cho, Chinnadhurai Sankar, Christopher Lin, 726 Kaushik Sadagopan, Shahin Shayandeh, Asli Celiky- 727 ilmaz, Jonathan May, and Ahmad Beirami. 2022. 728 Know thy strengths: Comprehensive dialogue state 729 tracking diagnostics. In Findings of the Association 730 for Computational Linguistics: EMNLP 2022, pages 731 5345–5359, Abu Dhabi, United Arab Emirates. As- 732 sociation for Computational Linguistics. 733
Hyung Won Chung, Le Hou, Shayne Longpre, Bar- 734 ret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi 735 Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 736 2022. Scaling instruction-finetuned language models. 737 arXiv preprint arXiv:2210.11416. 738
Daniel Deutsch, Rotem Dror, and Dan Roth. 2022. Re- 739 examining system-level correlations of automatic 740 summarization evaluation metrics. In Proceedings of 741 the 2022 Conference of the North American Chap- 742 ter of the Association for Computational Linguistics: 743 Human Language Technologies, pages 6038–6052, 744 Seattle, United States. Association for Computational 745 Linguistics. 746
Kaustubh D. Dhole, Varun Gangal, Sebastian 747 Gehrmann, Aadesh Gupta, Zhenhao Li, Saad Ma- 748 hamood, Abinaya Mahendiran, Simon Mille, Ashish 749 Srivastava, Samson Tan, Tongshuang Wu, Jascha 750 Sohl-Dickstein, Jinho D. Choi, Eduard Hovy, On- 751 drej Dusek, Sebastian Ruder, Sajant Anand, Na- 752 gender Aneja, Rabin Banjade, Lisa Barthe, Hanna 753 Behnke, Ian Berlot-Attwell, Connor Boyle, Car- 754 oline Brun, Marco Antonio Sobrevilla Cabezudo, 755 Samuel Cahyawijaya, Emile Chapuis, Wanxiang 756 Che, Mukund Choudhary, Christian Clauss, Pierre 757 Colombo, Filip Cornell, Gautier Dagan, Mayukh Das, 758 Tanay Dixit, Thomas Dopierre, Paul-Alexis Dray, Su- 759 chitra Dubey, Tatiana Ekeinhor, Marco Di Giovanni, 760 Rishabh Gupta, Rishabh Gupta, Louanes Hamla, 761 Sang Han, Fabrice Harel-Canada, Antoine Honore, 762
9
Ishan Jindal, Przemyslaw K. Joniak, Denis Kleyko,763 Venelin Kovatchev, Kalpesh Krishna, Ashutosh Ku-764 mar, Stefan Langer, Seungjae Ryan Lee, Corey James765 Levinson, Hualou Liang, Kaizhao Liang, Zhexiong766 Liu, Andrey Lukyanenko, Vukosi Marivate, Gerard767 de Melo, Simon Meoni, Maxime Meyer, Afnan Mir,768 Nafise Sadat Moosavi, Niklas Muennighoff, Timo-769 thy Sum Hon Mun, Kenton Murray, Marcin Namysl,770 Maria Obedkova, Priti Oli, Nivranshu Pasricha, Jan771 Pfister, Richard Plant, Vinay Prabhu, Vasile Pais,772 Libo Qin, Shahab Raji, Pawan Kumar Rajpoot, Vikas773 Raunak, Roy Rinberg, Nicolas Roberts, Juan Diego774 Rodriguez, Claude Roux, Vasconcellos P. H. S.,775 Ananya B. Sai, Robin M. Schmidt, Thomas Scialom,776 Tshephisho Sefara, Saqib N. Shamsi, Xudong Shen,777 Haoyue Shi, Yiwen Shi, Anna Shvets, Nick Siegel,778 Damien Sileo, Jamie Simon, Chandan Singh, Ro-779 man Sitelew, Priyank Soni, Taylor Sorensen, William780 Soto, Aman Srivastava, KV Aditya Srivatsa, Tony781 Sun, Mukund Varma T, A Tabassum, Fiona Anting782 Tan, Ryan Teehan, Mo Tiwari, Marie Tolkiehn,783 Athena Wang, Zijian Wang, Gloria Wang, Zijie J.784 Wang, Fuxuan Wei, Bryan Wilie, Genta Indra Winata,785 Xinyi Wu, Witold Wydmański, Tianbao Xie, Usama786 Yaseen, M. Yee, Jing Zhang, and Yue Zhang. 2021.787 Nl-augmenter: A framework for task-sensitive natu-788 ral language augmentation.789
Arash Einolghozati, Sonal Gupta, Mrinal Mohit, and790 Rushin Shah. 2019. Improving robustness of task ori-791 ented dialog systems. 3rd Conversational AI Work-792 shop at 33rd Conference on Neural Information Pro-793 cessing Systems.794
Guy Feigenblat, Chulaka Gunasekara, Benjamin Szna-795 jder, Sachindra Joshi, David Konopnicki, and Ranit796 Aharonov. 2021. TWEETSUMM - a dialog sum-797 marization dataset for customer service. In Find-798 ings of the Association for Computational Linguis-799 tics: EMNLP 2021, pages 245–260, Punta Cana, Do-800 minican Republic. Association for Computational801 Linguistics.802
Xiachong Feng, Xiaocheng Feng, and Bing Qin. 2021.803 A survey on dialogue summarization: Recent ad-804 vances and new frontiers. ArXiv, abs/2107.03175.805
Tim Fischer, Steffen Remus, and Chris Biemann. 2022.806 Measuring faithfulness of abstractive summaries. In807 Proceedings of the 18th Conference on Natural Lan-808 guage Processing (KONVENS 2022), pages 63–73,809 Potsdam, Germany. KONVENS 2022 Organizers.810
Jatin Ganhotra, Robert Moore, Sachindra Joshi, and811 Kahini Wadhawan. 2020. Effects of naturalistic vari-812 ation in goal-oriented dialog. In Findings of the Asso-813 ciation for Computational Linguistics: EMNLP 2020,814 pages 4013–4020, Online. Association for Computa-815 tional Linguistics.816
John Giorgi, Luca Soldaini, Bo Wang, Gary Bader, Kyle817 Lo, Lucy Lu Wang, and Arman Cohan. 2022. Explor-818 ing the challenges of open domain multi-document819 summarization. arXiv preprint arXiv:2212.10526.820
Bogdan Gliwa, Iwona Mochol, Maciej Biesek, and Alek- 821 sander Wawer. 2019. SAMSum corpus: A human- 822 annotated dialogue dataset for abstractive summa- 823 rization. In Proceedings of the 2nd Workshop on 824 New Frontiers in Summarization, pages 70–79, Hong 825 Kong, China. Association for Computational Linguis- 826 tics. 827
Prakhar Gupta, Cathy Jiao, Yi-Ting Yeh, Shikib Mehri, 828 Maxine Eskenazi, and Jeffrey Bigham. 2022. In- 829 structDial: Improving zero and few-shot general- 830 ization in dialogue through instruction tuning. In 831 Proceedings of the 2022 Conference on Empirical 832 Methods in Natural Language Processing, pages 505– 833 525, Abu Dhabi, United Arab Emirates. Association 834 for Computational Linguistics. 835
Hongyan Jing, Daniel Lopresti, and Chilin Shih. 2003. 836 Summarization of noisy documents: A pilot study. 837 In Proceedings of the HLT-NAACL 03 Text Summa- 838 rization Workshop, pages 25–32. 839
Taehee Jung, Dongyeop Kang, Lucas Mentch, and Ed- 840 uard Hovy. 2019. Earlier isn’t always better: Sub- 841 aspect analysis on corpus and system biases in sum- 842 marization. In Conference on Empirical Methods 843 in Natural Language Processing (EMNLP), Hong 844 Kong. 845
Vladimir Karpukhin, Omer Levy, Jacob Eisenstein, and 846 Marjan Ghazvininejad. 2019. Training on synthetic 847 noise improves robustness to natural noise in ma- 848 chine translation. In Proceedings of the 5th Work- 849 shop on Noisy User-generated Text (W-NUT 2019), 850 pages 42–47, Hong Kong, China. Association for 851 Computational Linguistics. 852
Seokhwan Kim, Yang Liu, Di Jin, Alexandros Papan- 853 gelis, Karthik Gopalakrishnan, Behnam Hedayatnia, 854 and Dilek Z. Hakkani-Tür. 2021. “how robust r u?”: 855 Evaluating task-oriented dialogue systems on spoken 856 conversations. 2021 IEEE Automatic Speech Recog- 857 nition and Understanding Workshop (ASRU), pages 858 1147–1154. 859
Kundan Krishna, Yao Zhao, Jie Ren, Balaji Lakshmi- 860 narayanan, Jiaming Luo, Mohammad Saleh, and Pe- 861 ter J Liu. 2022. Improving the robustness of sum- 862 marization models by detecting and removing input 863 noise. arXiv preprint arXiv:2212.09928. 864
Philippe Laban, Tobias Schnabel, Paul N Bennett, and 865 Marti A Hearst. 2022. SummaC: Re-visiting NLI- 866 based models for inconsistency detection in summa- 867 rization. Transactions of the Association for Compu- 868 tational Linguistics, 10:163–177. 869
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan 870 Ghazvininejad, Abdelrahman Mohamed, Omer Levy, 871 Veselin Stoyanov, and Luke Zettlemoyer. 2020. 872 BART: Denoising sequence-to-sequence pre-training 873 for natural language generation, translation, and com- 874 prehension. In Proceedings of the 58th Annual Meet- 875 ing of the Association for Computational Linguistics, 876 pages 7871–7880, Online. Association for Computa- 877 tional Linguistics. 878
10
Chin-Yew Lin. 2004. ROUGE: A package for auto-879 matic evaluation of summaries. In Text summariza-880 tion branches out, pages 74–81.881
Jiexi Liu, Ryuichi Takanobu, Jiaxin Wen, Dazhen Wan,882 Hongguang Li, Weiran Nie, Cheng Li, Wei Peng, and883 Minlie Huang. 2021. Robustness testing of language884 understanding in task-oriented dialog. In Proceed-885 ings of the 59th Annual Meeting of the Association886 for Computational Linguistics and the 11th Interna-887 tional Joint Conference on Natural Language Pro-888 cessing, pages 2467–2480, Online. Association for889 Computational Linguistics.890
Ailsa Meechan-Maddon. 2019. The effect of noise in891 the training of convolutional neural networks for text892 summarisation.893
Simon Mille, Kaustubh Dhole, Saad Mahamood, Laura894 Perez-Beltrachini, Varun Gangal, Mihir Kale, Emiel895 van Miltenburg, and Sebastian Gehrmann. 2021. Au-896 tomatic construction of evaluation suites for natural897 language generation datasets. In Thirty-fifth Con-898 ference on Neural Information Processing Systems899 Datasets and Benchmarks Track (Round 1).900
George A Miller. 1998. WordNet: An electronic lexical901 database. MIT press.902
Robert J. Moore and Raphael Arar. 2019. Conversa-903 tional UX Design: A Practitioner’s Guide to the Nat-904 ural Conversation Framework. Association for Com-905 puting Machinery, New York, NY, USA.906
Milad Moradi and Matthias Samwald. 2021. Evaluating907 the robustness of neural language models to input908 perturbations. In Conference on Empirical Methods909 in Natural Language Processing.910
Xing Niu, Prashant Mathur, Georgiana Dinu, and Yaser911 Al-Onaizan. 2020. Evaluating robustness to input912 perturbations for neural machine translation. In Pro-913 ceedings of the 58th Annual Meeting of the Asso-914 ciation for Computational Linguistics, pages 8538–915 8544, Online. Association for Computational Lin-916 guistics.917
Baolin Peng, Chunyuan Li, Zhu Zhang, Chenguang918 Zhu, Jinchao Li, and Jianfeng Gao. 2020. Rad-919 dle: An evaluation benchmark and analysis plat-920 form for robust task-oriented dialog systems. ArXiv,921 abs/2012.14666.922
Denis Peskov, Joe Barrow, Pedro Rodriguez, Graham923 Neubig, and Jordan Boyd-Graber. 2019. Mitigating924 noisy inputs for question answering. arXiv preprint925 arXiv:1908.02914.926
Lihua Qian, Lin Qiu, Weinan Zhang, Xin Jiang, and927 Yong Yu. 2019. Exploring diverse expressions928 for paraphrase generation. In Proceedings of the929 2019 Conference on Empirical Methods in Natu-930 ral Language Processing and the 9th International931 Joint Conference on Natural Language Processing932 (EMNLP-IJCNLP), pages 3173–3182, Hong Kong,933 China. Association for Computational Linguistics.934
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine 935 Lee, Sharan Narang, Michael Matena, Yanqi Zhou, 936 Wei Li, and Peter J Liu. 2020a. Exploring the limits 937 of transfer learning with a unified text-to-text trans- 938 former. The Journal of Machine Learning Research, 939 21(1):5485–5551. 940
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine 941 Lee, Sharan Narang, Michael Matena, Yanqi Zhou, 942 Wei Li, Peter J Liu, et al. 2020b. Exploring the 943 limits of transfer learning with a unified text-to-text 944 transformer. Journal of Machine Learning Research, 945 21(140):1–67. 946
Harvey Sacks, Emanuel A. Schegloff, and Gail D. Jef- 947 ferson. 1974. A simplest systematics for the orga- 948 nization of turn-taking for conversation. Language, 949 50:696 – 735. 950
Sailik Sengupta, Jason Krone, and Saab Mansour. 2021. 951 On the robustness of intent classification and slot la- 952 beling in goal-oriented dialog systems to real-world 953 noise. In Proceedings of the 3rd Workshop on Nat- 954 ural Language Processing for Conversational AI, 955 pages 68–79, Online. Association for Computational 956 Linguistics. 957
Xin Tian, Xinxian Huang, Dongfeng He, Yingzhan Lin, 958 Siqi Bao, H. He, Liankai Huang, Qiang Ju, Xiyuan 959 Zhang, Jianyue Xie, Shuqi Sun, Fan Wang, Hua Wu, 960 and Haifeng Wang. 2021. Tod-da: Towards boosting 961 the robustness of task-oriented dialogue modeling on 962 spoken conversations. ArXiv, abs/2112.12441. 963
Vaibhav Vaibhav, Sumeet Singh, Craig Stewart, and Gra- 964 ham Neubig. 2019. Improving robustness of machine 965 translation with synthetic noise. In Proceedings of 966 the 2019 Conference of the North American Chap- 967 ter of the Association for Computational Linguistics: 968 Human Language Technologies, Volume 1 (Long and 969 Short Papers), pages 1916–1920, Minneapolis, Min- 970 nesota. Association for Computational Linguistics. 971
Xuezhi Wang, Haohan Wang, and Diyi Yang. 2022. 972 Measure and improve robustness in NLP models: A 973 survey. In Proceedings of the 2022 Conference of 974 the North American Chapter of the Association for 975 Computational Linguistics: Human Language Tech- 976 nologies, pages 4569–4586, Seattle, United States. 977 Association for Computational Linguistics. 978
Larry Wasserman. 2004. All of statistics: a concise 979 course in statistical inference, volume 26. Springer. 980
Linzi Xing, Wen Xiao, and Giuseppe Carenini. 2021. 981 Demoting the lead bias in news summarization via 982 alternating adversarial learning. In Annual Meeting 983 of the Association for Computational Linguistics. 984
Alex Yorke. butter-fingers. 985
Lin Yuan and Zhou Yu. 2019. Abstractive dialog sum- 986 marization with semantic scaffolds. arXiv preprint 987 arXiv:1910.00825. 988
11
Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Pe-989 ter J. Liu. 2019. Pegasus: Pre-training with extracted990 gap-sentences for abstractive summarization.991
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q.992 Weinberger, and Yoav Artzi. 2020. BERTScore:993 Evaluating text generation with BERT. In Interna-994 tional Conference on Learning Representations.995
Lulu Zhao, Fujia Zheng, Keqing He, Weihao Zeng, Yue-996 jie Lei, Huixing Jiang, Wei Wu, Weiran Xu, Jun Guo,997 and Fanyu Meng. 2021. Todsum: Task-oriented di-998 alogue summarization with state tracking. ArXiv,999 abs/2110.12680.1000
Qi Zhu, Zheng Zhang, Yan Fang, Xiang Li, Ryuichi1001 Takanobu, Jinchao Li, Baolin Peng, Jianfeng Gao,1002 Xiaoyan Zhu, and Minlie Huang. 2020. Convlab-2:1003 An open-source toolkit for building, evaluating, and1004 diagnosing dialogue systems. In Annual Meeting of1005 the Association for Computational Linguistics.1006
Caleb Ziems, Jiaao Chen, Camille Harris, Jes-1007 sica Brooke Anderson, and Diyi Yang. 2022. Value:1008 Understanding dialect disparity in nlu. ArXiv,1009 abs/2204.03031.1010
A Appendix 1011
A.1 Details/Examples of Perturbations 1012
See Table 6. 1013
A.2 Details of annotation guidelines of quality 1014 validation in §5.2 1015
For annotation collection, we only allowed anno- 1016 tators proficient in English from a small group of 1017 the most experienced annotators adjudicated by the 1018 Appen platform; from any country. We also used 1019 hidden test questions for quality control and re- 1020 quired annotators to maintain at least 80% accuracy 1021 throughout the job on these hidden test questions. 1022 These test questions are pre-labeled and are used 1023 before and during the task to quiz the annotator. We 1024 selected 15 test questions from the validation split 1025 of each dataset ensuring that these questions do not 1026 overlap with questions seen by the annotators for 1027 the actual annotation task. Figure 4 shows the an- 1028 notation guidelines and Figure 5 shows examples 1029 provided for this task. 1030
A.3 Details of annotation guidelines for the 1031 validity of trends in §5.6 1032
Quality Control: For this task, as well we only 1033 allowed annotators proficient in English from a 1034 small group of the most experienced annotators 1035 adjudicated by the Appen platform; from any coun- 1036 try. We also used hidden test questions for quality 1037 control and required annotators to maintain at least 1038 80% accuracy throughout the job on these hidden 1039 test questions. Figure 6 shows the annotation guide- 1040 lines, and Figure 7 shows examples provided for 1041 this task. 1042
Number of annotations: In the main task, each 1043 annotator was shown 5 examples per page with one 1044 hidden test example. For each example, we col- 1045 lected three annotations. In cases where there was 1046 no agreement among the initial three annotations, 1047 we obtained additional annotations. A maximum 1048 of five annotations was considered. 1049
Noise Filtering: Before computing consistency 1050 scores, we took several steps to filter out noisy an- 1051 notations. The Appen platform estimates the trust 1052 score for each worker (by calculating accuracy on 1053 hidden test examples) and also marks examples 1054 as tainted if it is annotated by an annotator whose 1055 accuracy score has fallen below the minimum accu- 1056 racy threshold. To retain only the highest quality an- 1057 notations, we remove annotations that were marked 1058 as tainted and only keep annotations from workers 1059
12
whose trust score is 100%. On qualitatively exam-1060 ining the annotations we also found cases where1061 the two summaries were word-by-word the same,1062 yet the annotator did not give a rating of 4 (highly1063 similar or exact match). Since this is a case of ob-1064 vious noise, we remove such cases. If an example1065 has less than 3 annotations left after the filtering1066 step, we drop the example. After this filtering, we1067 finally use 314 annotations to conduct our analysis.1068
A.4 Targeted dialogue perturbations to1069 investigate the repetition bias1070
To delve deeper into the repetition bias observed in1071 the models, we conducted targeted perturbations,1072 where we repeat utterances based on whether the in-1073 formation conveyed in those utterances was consid-1074 ered important by the reference summary. Specif-1075 ically, we identify utterances that are highly rele-1076 vant and least relevant to the reference summary.1077 To measure relevance, we compute semantic simi-1078
larity8 between each utterance and each sentence 1079 in the reference summary. For each summary sen- 1080 tence, we then determine the most (least) relevant 1081 utterance by selecting the one with the highest (low- 1082 est) similarity with the summary sentence. When 1083 perturbing the most relevant utterance, we perturb 1084 the utterances that were identified as relevant to 1085 at least one summary sentence. When perturbing 1086
8using sentence transformers [CITE]
13
the least relevant utterance, we perturb the utter-1087 ances that were identified as least relevant to all the1088 summary sentences.1089
As shown in Table 7, we observe that the model1090 exhibits the highest change in saliency scores when1091 we perturb the least relevant utterance, which fur-1092 ther demonstrates the model’s tendency to consider1093 repeated information as important, even though it1094 was not considered important as per the reference1095 summary. In contrast, repetition of the most rele-1096 vant utterance shows the least change in the scores,1097 since the model already focuses on the most rele-1098 vant information before perturbation and after re-1099 peating that utterance, it still remains important to1100 be included in the summary.1101
A.5 Sensitivity to perturbation rate 1102 A.6 Perturbation-wise impact on zero-shot 1103 models 1104 See Table 8 and Table 9 1105
A.7 Correlation analysis 1106 Table 10 shows the Pearson correlations between 1107 pairs of dimensions on the TweetSum dataset. Cor- 1108 relations scores are also visualized in Figures 10, 1109 11, 12. Similar correlation are also observed on 1110 SAMSum (Figures 14, 15, 13) and TODSum datasets 1111 (Figures 17, 18, 16). 1112
A.8 Analysis using ROUGE-L and SummaC 1113 scores 1114
14
15
16
17

heading: Future Work
text: Overall, our work highlights030 robustness challenges in current open models031 and provides insights for future research. 032
1 Introduction033
Real-life conversations often exhibit a wide range034 of language variations, including typographical er-035 rors, grammatical mistakes, and certain exchanges036 such as repetitions and speaker interruptions, which037 are unrelated to the primary purpose of the con-038 versation (Sacks et al. , 1974). However, existing039 dialogue summarization datasets, which are used040 to train current summarization models, do not ad-041 equately capture these variations, as they are typi-042 cally constructed by annotators simulating specific043
scenarios (Yuan and Yu, 2019) or extracted from 044 English-speaking practice websites (Gliwa et al. , 045 2019). Even the datasets consisting of real-life 046 conversations (Feigenblat et al. , 2021) can exhibit 047 only a limited range of variations owing to practi- 048 cal limitations posed by the data collection process 049 (e. g. , high or low prevalence of conversations from 050 different social demographics). Consequently, dia- 051 logue summarization models deployed in business 052 scenarios encounter diverse variations not observed 053 during training. This raises a crucial question: Can 054 current dialogue summarization models effectively 055 handle conversations with naturally occurring vari- 056 ations that are legitimate inputs but not observed in 057 the training data? 058
In this work, we study the impact of naturally 059 occurring variations on the performance of the 060 state-of-the-art open dialogue summarization mod- 061 els (with publicly known architecture, weights, 062 and training corpus) using three publicly avail- 063
1 56
able datasets. We examine the performance of064 encoder-decoder Transformer models in two setups065 a) fine-tuned on specific dialogue summarization066 datasets (Lewis et al. , 2020; Zhang et al. , 2019;067 Raffel et al. , 2020b), and b) instruction-tuned mod-068 els which have shown impressive zero-shot perfor-069 mance more recently (Gupta et al. , 2022; Chung070 et al. , 2022). Such models are often preferred071 in high-stakes business settings (e. g. , medical, le-072 gal, and customer support) over proprietary models073 (e. g. , ChatGPT), owing to user privacy concerns. 074
To simulate variations we design two kinds of075 perturbations: (a) utterance-level perturbations,076 and (b) dialogue-level perturbations (defined in077 Section 3), which are inspired by common real-life078 interaction patterns from the Natural Conversation079 Framework (Moore and Arar, 2019). We evaluate080 the performance of summarization models along081 three conceptually distinct robustness dimensions—082 consistency, saliency, and faithfulness—and elabo-083 rate on their empirical relationship. 084
Our analysis reveals that both fine-tuned and085 instruction-tuned models are impacted by utterance086 and dialogue-level perturbations. Instruction-tuned087 models are impacted more than fine-tuned models088 and are also more susceptible to dialogue-level per-089 turbations than utterance-level perturbations. Both090 types of models show a preference for information091 from repeated, long, and leading utterances in the092 dialogue. Figure 1 shows an example where the093 model includes repeated utterances in the summary,094 whereas the non-repeated original utterance wasn’t095 included in the summary before perturbation. We096 also validate our findings via human evaluation. 097
Finally, we investigate whether fine-tuned mod-098 els improve by training with perturbed data. We099 find that this approach does not consistently en-100 hance performance, and different perturbations re-101 quire varying amounts of training examples for102 gains. Thus, further research is needed to address103 these robustness challenges. 104
2 Related Work105
Prior work has investigated the robustness of lan-106 guage understanding models mainly focusing on107 classification tasks (Moradi and Samwald, 2021). 108 Some dialogue-related classification tasks have109 also been explored, including dialogue act predic-110 tion (Liu et al. , 2021), intent detection and slot111 tagging (Einolghozati et al. , 2019; Sengupta et al. ,112 2021), state tracking and dialogue modeling (Cho113 et al. , 2022; Tian et al. , 2021; Zhu et al. , 2020; Kim114 et al. , 2021; Peng et al. , 2020). 115
Some studies have also investigated the robust- 116 ness of neural language generation models, includ- 117 ing neural machine translation (Niu et al. , 2020; 118 Karpukhin et al. , 2019; Vaibhav et al. , 2019), ques- 119 tion answering (Peskov et al. , 2019), and open do- 120 main multi-document summarization (Giorgi et al. , 121 2022). However, some of these studies consider 122 perturbations that are of extreme nature (e. g. , ran- 123 dom shuffling and deletion of words) and may oc- 124 cur rarely in the real world. Ganhotra et al. (2020) 125 investigated the impact of natural variations on re- 126 sponse prediction tasks in goal-oriented dialogues. 127
For summarization task in particular, previous 128 studies focused on summarizing news articles and 129 documents (Jing et al. , 2003; Meechan-Maddon, 130 2019; Krishna et al. , 2022). However, the nature 131 of noise in a multi-party dialogue differs signifi- 132 cantly from noise in documents. While some types 133 of noise (e. g. , spelling mistakes, grammatical er- 134 rors) could occur in both, the patterns such as rep- 135 etitions, reconfirmations, hesitations, and speaker 136 interruptions (Sacks et al. , 1974; Feng et al. , 2021; 137 Chen and Yang, 2021) are peculiar to dialogues, 138 posing unique challenges for accurate and robust 139 summarization. The focus of this work is to assess 140 the robustness of dialogue summarization models 141 in the presence of naturally occurring variations, 142 which has been understudied in the prior literature. 143
3 Simulating Naturally Occurring 144 Variations 145
To introduce naturally occurring variations in con- 146 versations, we consider two kinds of simulated 147 perturbations, utterance-level and dialogue-level. 148 We apply each perturbation individually to a dia- 149 logue to study its impact systematically. Our per- 150 turbations are inspired by the Natural Conversation 151 Framework (Moore and Arar, 2019), created after 152 analyzing real-world conversations across various 153 use cases and provides common interactive pat- 154 terns that occur in real life. 1 Appendix A. 1 lists 155 examples for each perturbation. 156
3. 1 Utterance-level Perturbations 157 The utterance-level perturbations modify a single 158 utterance and are adapted from (Liu et al. , 2021). 159 We perturb each utterance of the dialogue. For per- 160 turbations where multiple words in an utterance 161 can be perturbed (e. g. , spelling mistake, character 162 casing), we consider only low-modification levels 163 (i. e. , perturb a word with 0. 2 probability), which 164
1Some examples include patterns such as C1. 0 (opening greeting agent), C4. 6 (closing success check), B2. 1. 0 (repeat request), A2. 8 (hold request). 2
also cause a considerable change in model perfor-165 mance. 2166
Typographical Errors Typographical errors oc-167 cur when participants try to type quickly in chat-168 based interactions. We use simple regex-based per-169 turbations, e. g. , punctuation marks removal, whites-170 pace removal or addition, changing letter casing,171 and substitutions of common expansions and con-172 tractions. We introduce spelling errors following173 the approach of Yorke as used in (Mille et al. , 2021),174 replacing random letters with other letters closely175 co-located on the keyboard positions. We ensure176 that mistakes are not introduced in a proper-noun177 phrase (e. g. , restaurant name) to avoid changes in178 important information. 179
Grammatical Errors We focus on two frequent180 grammatical errors: dropping determiners and181 subject-verb disagreements. To drop determiners,182 we drop all the words in a sentence with the DET183 tag. To introduce subject-verb disagreement, we184 identify auxiliary verbs (via AUX tag) and convert185 between plural and singular forms as appropriate,186 keeping the tense unchanged. 187
Language-use Variations Users can vary in their188 choices of dialect and vocabulary. We consider189 three language-use perturbations: substituting ad-190 jectives with synonyms, inflectional variations, and191 synthetic African American Vernacular English192 (AAVE) dialect. For synonym substitution, we sub-193 stitute adjectives in an utterance with their WordNet194 (Miller, 1998) synonyms. To introduce inflectional195 variations, we follow the approach proposed in196 Dhole et al. (2021), where we lemmatize each con-197 tent word in an utterance, randomly sample a valid198 POS category, and re-inflect the word according to199 the chosen category. To transform an utterance to200 synthetic AAVE dialect, we use the set of lexical201 and morphosyntactic transformation rules proposed202 by Ziems et al. (2022). 203
3. 2 Dialogue-level Perturbations204
We introduce new utterances that contribute no205 additional information, to test a model’s ability to206 focus on the overall meaning of a conversation and207 identify salient information. 208
Repetitions Repeating and rephrasing occur209 commonly in real-life spoken conversations. In210 this perturbation, we randomly select an utterance211
2See Appendix A. 5 for analysis with different perturbation rates. to repeat. 3 We then inject a synthetic utterance 212 requesting the other participant to repeat the in- 213 formation (e. g. , ‘Sorry, I couldn’t hear you, can 214 you repeat?’). 4 Since humans tend to rephrase the 215 original message slightly instead of repeating it 216 verbatim, we paraphrase the original utterance be- 217 fore including it as a response to the request for 218 repetition. We use Qian et al. (2019)’s paraphraser 219 for this task. The rest of the dialogue remains un- 220 changed. This perturbation allows us to examine 221 repetition bias; i. e. , does the model consider re- 222 peated utterances more significant, even when they 223 do not contain important information? 224
Time delays A participant may ask the other 225 party to wait while they gather information. To 226 simulate this, we add three synthetic utterances 227 consecutively: a request to wait (e. g. , ‘Just give 228 me a few minutes. ’), an acknowledgment from the 229 other participant (e. g. , ‘Sure’), and an expression 230 of gratitude from the first participant (e. g. , ‘Thanks 231 for waiting. ’). These utterances are inserted after 232 a randomly selected utterance from the participant 233 being asked to wait. 234
Greeting and closing remarks It is also com- 235 mon to begin a conversation with a friendly greet- 236 ing and end with some closing remarks. For the 237 greetings perturbation, we insert a greeting as the 238 first utterance, such as ‘Hi! I am your customer 239 support assistant. How may I help you today?’ in 240 customer support dialogues and ‘Hey there!’ in 241 open-domain chit-chat. For the closing remarks 242 perturbation, we insert a final message: ‘Thank 243 you for contacting us. ’ in customer support dia- 244 logues and ‘Cool, talk to you later!’ in open domain 245 chit-chat. Each perturbation is applied individually 246 to a dialogue. Both of these perturbations help 247 us investigate structural biases present in dialogue 248 summarization models, also known to impact news 249 summarization models (Xing et al. , 2021; Jung 250 et al. , 2019). For instance, the greeting perturba- 251 tion helps examine lead bias (preference for the 252 first utterance), and closing remarks perturbation 253 helps examine recency bias (preference for the last 254 utterance). 255
3See Appendix A. 4 for targeted perturbations, where we select an utterance to repeat based on its saliency. 4We use this utterance to operationalize the repetition perturbation, inspired by spoken dialogues. However, repetitions can also appear in written dialogues (e. g. , sending the same message multiple times to ensure communication, emphasizing points, or dealing with technical issues. ). Furthermore, models trained on written dialogues are often deployed to summarize transcripts of spoken dialogues, where such utterances are more common. 3
Split and combined utterances In chat-based256 conversations, participants can have varying prefer-257 ences for either conveying information over multi-258 ple consecutive utterances or sending one long mes-259 sage. To simulate split utterance perturbation, we260 divide a randomly sampled utterance into consec-261 utive utterances by splitting it at every five words. 262 Conversely, to simulate combined utterance per-263 turbation, we identify sequences of consecutive264 utterances from a single participant in a dialogue265 and concatenate them. We combine consecutive ut-266 terances from only one participant at a time. Each267 perturbation is applied individually to a dialogue. 268 Both these perturbations allow us to examine long269 bias (the model’s preference to include a long utter-270 ance over shorter utterances, even when multiple271 short utterances include salient information). 272
3. 3 Quality evaluation of perturbed dialogues273 We conduct a human validation of the perturbed274 dialogues. The goal of this evaluation is to ensure275 that our perturbations do not alter the dialogue’s276 meaning or introduce new information, thereby277 validating the quality of our perturbed test set. We278 sample 20 dialogues and their summaries from each279 of the three datasets (§5. 1) and perturb each dia-280 logue with all of the utterance and dialogue-level281 perturbations, resulting in a total of 480 dialogues. 282 Two annotators are asked to determine whether283 the reference summary for the original dialogue re-284 mains valid for all the perturbed dialogues (see Ap-285 pendix A. 2 for details on annotation guidelines). In286 cases of disagreement, a third annotator breaks the287 tie. The annotators marked 97. 5% of the perturbed288 dialogues as being reasonably summarized by the289 summary of the original dialogue, thus validating290 the use of proposed perturbations to investigate the291 robustness of dialogue summarization models. Our292 human evaluation also suggests that our perturba-293 tions do not drastically alter the dialogue and the294 dialogues remain readable and semantically con-295 sistent. Otherwise, for an altered dialogue, the296 original summary would have been marked invalid. 297
4 Quantifying Robustness298
For tasks involving text generation, such as di-299 alogue summarization, measuring robustness in-300 volves determining the relationship between differ-301 ent pairs of natural language texts. As a result, the302 robustness of generative tasks is less well-defined,303 compared to a classification task (Liu et al. , 2021)304 and can manifest in several ways. We consider305 three dimensions for measuring robustness issues306 that can arise in dialogue summarization. 307
Let x denote the original dialogue, yr be the 308 reference summary of the original dialogue, f be 309 the summarization model trained on (x, yr) ∼ D, 310 and f(x) be its prediction over x. Let x′ = x + 311 δ denote the perturbed dialogue and f(x′) be its 312 predicted summary. 313
Consistency A model is consistent (and hence 314 robust) under a perturbation (δ) if the two sum- 315 maries, f(x) and f(x′ = x+ δ), are semantically 316 similar, resulting in minimal change. We quantify 317 the change in model-generated output as follows, 318
∆zc = |SCORE(f(x), f(x))− SCORE(f(x), f(x′))|
SCORE(f(x), f(x)) (1) 319
further simplified as, 320
∆zc = 1− SCORE(f(x), f(x′)) (2) 321
where SCORE is any text similarity metric (e. g. , 322 BERTScore) that assigns a value of 1 for identical 323 inputs and 0 for dissimilar inputs. By definition, 324 ∆zc ∈ [0, 1]. Note that consistency is sufficient but 325 not necessary for robustness: a good summary can 326 be expressed in diverse ways, which leads to high 327 robustness but low consistency. 328
Saliency Assuming that the reference summary 329 includes the most salient information conveyed 330 in the input dialogue, we compute the change 331 in salient information captured by the model- 332 generated summaries (before and after perturba- 333 tion) w. r. t the reference summary as follows: 334
∆zs = |SCORE(yr, f(x))− SCORE(yr, f(x′))|
SCORE(yr, f(x)) (3) 335
where SCORE is any text similarity metric (e. g. , 336 BERTScore). Since ∆zs measures the normalized 337 change in similarity scores, ∆zs ∈ [0, 1]. 338 Faithfulness Faithfulness refers to the extent 339 to which the generated summary is supported by 340 the content of the input dialogue, thus accurately 341 reflecting the information without introducing spu- 342 rious or fabricated details, commonly termed as 343 hallucinations. We compute the change in faithful- 344 ness as follows: 345
∆zf = |SCORE(x, f(x))− SCORE(x, f(x′))|
SCORE(x, f(x)) (4) 346
where SCORE is any text-based precision met- 347 ric measuring the fraction of information in the 348 summary (f(x)) supported by the input dialogue 349
4
(x) (e. g. , BERTScore-Precision). Since ∆zf mea-350 sures the normalized change in precision scores,351 ∆zf ∈ [0, 1]. Note that, the second term in the352 numerator compares x with f(x′) since we are353 interested in measuring the fraction of summary354 information supported by the ‘original dialogue. ’355 Furthermore, since our added perturbations do not356 add any new information to the dialogue, x and357 x′ would essentially contain the same information. 358 Clearly, for all three dimensions, the higher the ∆z,359 the lower the robustness of the model. 360
5 Evaluating Robustness361
We present our key observations on how various362 perturbations impact the model performance. 363
5. 1 Implementation Details364 Datasets We consider two task-oriented dia-365 logues, TWEETSUMM (Feigenblat et al. , 2021) and366 TODSum (Zhao et al. , 2021), both consisting of367 conversations between an agent and a customer. 368 TODSum comprises dialogues from multiple sub-369 domains (restaurants, movies, etc), collected via370 crowdsourcing where annotators are tasked to gen-371 erate dialogues based on a given scenario. In con-372 trast, TWEETSUMM focuses solely on customer sup-373 port conversations occurred at Twitter. We also374 include SAMSUM (Gliwa et al. , 2019), a corpus of375 chit-chat dialogues between two or more friends. 376
Models We analyze the robustness of three Trans-377 former based encoder-decoder models for dialogue378 summarization, Pegasus-large (568M parame-379 ters) (Zhang et al. , 2019), BART-large (400M380 parameters) (Lewis et al. , 2020) and T5-base381 (220M parameters) (Raffel et al. , 2020a), whose382 details are publicly available. All models have383 a comparable number of parameters. We fine-384 tune each model on the train split of the respec-385 tive dataset. We use beam search5 with size 5 to386 generate summaries. We also investigate the robust-387 ness of instruction-tuned versions of two of these388 models, DIAL-BART0 (406M parameters) (Gupta389 et al. , 2022) and FLAN-T5-large (783M parame-390 ters) (Chung et al. , 2022), used as zero-shot sum-391 marizers, without fine-tuning on the three dialogue392 summarization datasets considered in this work. 393
Metrics We evaluate summaries using394 BERTScore (Zhang et al. , 2020), which has395 been shown to better correlate with human judg-396 ment (Fischer et al. , 2022). BERTScore calculates397 precision, recall, and F1 scores by comparing a398
5Nucleus sampling omitted to avoid sampling variance. model-generated summary to a reference summary. 399 We use F1 to compute consistency and saliency, 400 and precision to compute faithfulness. To 401 validate observed trends, we additionally evaluate 402 summaries using ROUGE-L metric (Lin, 2004), 403 which measures lexical overlap, and SummaC 404 metric (Laban et al. , 2022), which measures 405 factual consistency. For all the reported results, we 406 observe similar trends via ROUGE-L and SummaC 407 (Tables 11,12,13 in Appendix A. 8). While we 408 report results using these metrics, the three 409 robustness dimensions can be computed using 410 any evaluation metric. For each reported result, 411 we use a non-parametric bootstrap (Wasserman, 412 2004, ch. 8) to infer confidence intervals (CIs). We 413 utilize 104 bootstrap samples of the dialogues to 414 report 95% bootstrap CIs via the normal interval 415 method (Wasserman, 2004, ch. 8. 3). 416
5. 2 How robust are fine-tuned models? 417 Fine-tuned dialogue summarization models are 418 affected by both utterance and dialogue level 419 perturbations Table 1 shows the change in con- 420 sistency, saliency, and faithfulness owing to utter- 421 ance and dialogue level perturbations on all three 422 datasets. All three models are equally impacted by 423 various perturbations. Models trained on TweetSum 424 and SAMSum are impacted equally by both utterance- 425 level and dialogue-level perturbations. TODSum 426 is the least impacted, since this dataset contains 427 template-based summaries where only entities from 428 the dialogue are required to be filled. We see a ma- 429 jor impact on faithfulness, with the highest impact 430 on the model trained on the TODSum dataset. 431
Impact of utterance perturbations Table 2 432 shows that these perturbations have a compara- 433 ble impact (shown averaged over all three models). 434 Models trained on TODSum exhibit little change in 435 consistency and saliency, but a significant change 436 in faithfulness. This is expected since the TODSum 437 summaries are extractive, following a pre-defined 438 template, and only require substituting entity infor- 439 mation extracted from the dialogue. Since the tem- 440 plate is fixed and the summaries can only change 441 in entity information before and after perturbation 442 and w. r. t reference summary, we see a small change 443 in consistency and saliency. However, we observe 444 a large change in faithfulness, as this dimension 445 focuses on the factual correctness of the summary. 446
Impact of dialogue perturbations: Table 3 re- 447 ports the impact of dialogue-level perturbations 448 (averaged over all models) and shows significant 449 changes for repetition, time delays, greetings, and 450
5
split utterances. For instance, when subjected to451 repetitions, the models tend to include repeated452 utterances in the summary, even if they were pre-453 viously deemed unimportant (repetition bias; Fig-454 ure 1). Additionally, the models demonstrate a455 preference for the first utterance in a dialogue (lead456 bias), rendering them susceptible to greetings per-457 turbation. This observation aligns with prior find-458 ings for news summarization, where sentences at459 the beginning of an article are more likely to con-460 tain summary-worthy information. Similarly, in461 customer-support conversations, the first utterance462 frequently addresses the primary issue faced by the463 customer. Consequently, models trained on such464 datasets exhibit lead bias. Finally, the models pre-465 fer lengthy utterances in the summary (long bias),466 by being more affected by split perturbations, and467 less affected by short utterances combined. 468
5. 3 Effect of model size on robustness469 Table 4 shows the change in consistency for models470 with different number of parameters: BART-base,471 BART-large, T5-base, and T5-small. The mod-472 els are almost equally affected by perturbations,473 irrespective of size, suggesting that robustness is-474 sues cannot be mitigated by scaling the model size. 475
5. 4 How robust are instruction-tuned models476 when used as zero-shot summarizers?477
DIAL-BART0 and FLAN-T5-large are instruction-478 tuned on multiple tasks, with DIAL-BART0, in479 particular, is instruction-tuned on dialog-specific480 tasks. However, neither model was trained on the481 TweetSum dataset, providing a zero-shot setting482
to evaluate their dialogue summarization capabil- 483 ities. As depicted in Table 5, both DIAL-BART0 484 (∆zc=30. 37% for utterance and 34. 30% for dia- 485 logue) and FLAN-T5 (∆zc=38. 23% for utterance 486 and 44. 12% for dialogue) are much more sensitive 487 to perturbations compared to their fine-tuned coun- 488 terparts (∆zc=17. 36% for utterance and 16. 82% 489 for dialogue, averaged over three models). 490
In contrast to fine-tuned models, the zero-shot 491 models are affected more by the dialogue-level 492 perturbations (∆zc=34. 30% for DIAL-BART0 and 493 ∆zc=44. 12% for FLAN-T5) than utterance-level 494 perturbations (∆zc=30. 37% for DIAL-BART0 and 495 ∆zc=38. 23% for FLAN-T5). Among utterance-level 496 perturbations, similar to the fine-tuned models, 497 zero-shot models are also impacted equally by all 498 perturbations. Among dialogue-level perturbations 499 as well, similar to the fine-tuned models, zero-shot 500 models are most impacted by repetitions, greetings 501 and split utterances (Appendix A. 6). 502
We additionally consider a recent instruction- 503 tuned large language model, Llama-2-70B, with 504 only publicly available weights. This model is also 505 significantly larger (70B) than the other models 506 (<0. 9B). Our results show high sensitivity to per- 507 turbations for this model (∆zc=47. 10% for utter- 508 ance and ∆zc=54. 53% for dialogue perturbations), 509 though we leave detailed human evaluation of the 510 outputs of this model for future work. 511
5. 5 Validity of findings with human evaluation 512
We conduct another human evaluation to con- 513 firm the trends observed with automatic similar- 514 ity metrics. Specifically, we collect similarity 515 scores between summary pairs using human an- 516 notations instead of automated similarity metrics 517 (e. g. , BERTScore). The goal is to ensure that ro- 518 bustness trends observed with automated metrics 519 are similar to those from human evaluation. 520
We use the consistency dimension for this eval- 521 uation for two main reasons: 1) Empirically, the 522 three robustness dimensions exhibit a strong cor- 523 relation (Table 10). Thus, using any of the three 524
6
dimensions would suffice for human evaluation,525 and (2) Among the three dimensions, consistency526 is easiest to use for human evaluation since it only527 requires the comparison of two summaries. 528
We collected annotations via the Appen platform529 (https://appen. com/), asking annotators to com-530 pare summaries of the perturbed and unperturbed531 dialogue, ranking their similarity on a Likert scale532 of 1 (dissimilar) to 4 (identical or paraphrases). To533 collect annotations, we used the same set of 20 dia-534 logues as in §3. 3 from the TweetSum dataset. Each535 dialogue was perturbed with one of the eight cate-536 gories (utterance- and dialogue-level), yielding 160537 summary pairs to be annotated. 538
We collected 3 annotations per summary pair,539 totaling 480 annotations; after filtering out noisy540 annotations, we conducted our analysis on the re-541 maining 314 examples (Appendix A. 3 provides an-542 notation procedure and guidelines). We aggregate543 annotations using majority voting to get similar-544 ity scores. To compute consistency scores (equa-545 tion 1), we map the Likert scale to continuous nu-546 meric scores from 0 to 1. We compute mean scores547 across all pairs for a given dataset and perturbation. 548
As shown in Figure 2, we observe similar trends,549 with models exhibiting repetition, long, and lead550 biases, and that models are affected nearly equally551 by all utterance perturbations. While the absolute552 values of ∆zc differ between calculations using au-553 tomatic metrics and human annotations, the relative554 impact of different perturbations on the model is555 similar. For instance, combined utterances and clos-556 ing remarks have the least impact than repetition,557 greetings, and split utterance perturbations. 6558
6Except time delays, owing to noise in human annotations. 5. 6 Relationship among dimensions 559
While theoretically, three dimensions (§4) measure 560 different aspects of robustness, empirically they ex- 561 hibit a strong correlation of > 84% across datasets 562 and models (details in Table 10 in Appendix). 563
This observation can be conceptually explained 564 to some extent. For instance, high saliency implies 565 high consistency: if summaries before and after per- 566 turbation are similar to the reference summary, they 567 will be similar to each other, leading to low ∆zs 568 and thus low ∆zc. Similarly, high saliency implies 569 high faithfulness: if the model-generated summary 570 is similar to the reference summary, it will also be 571 factually consistent with the input dialogue, lead- 572 ing to low ∆zs and thus low ∆zf. However, if 573 ∆zs is large, the model could remain faithful un- 574 der perturbation (small ∆zf ): summaries can be 575 different from the reference summary yet consis- 576 tent with the input dialogue. Thus, conceptually, 577
7
the relation can be explained in only one direction,578 but empirically the dimensions are highly corre-579 lated. Nevertheless, our findings are insightful in580 their own right, suggesting that the high correlation581 among all dimensions could be valuable for future582 robustness studies. For instance, the consistency583 or faithfulness dimension can serve as reference-584 free measures of robustness. Consistency is also585 the easiest to use for human evaluation, as it only586 requires comparing two summaries. 587
6 Improving Robustness588
One solution to address robustness issues could be589 to employ reverse heuristics to remove perturba-590 tions from dialogues. However, not all perturba-591 tions can be easily discovered and removed. For592 example, in repetition or time delay perturbations,593 the repeated utterance may include less information594 or be paraphrased compared to the original. While595 greetings and closing remarks might be simpler596 to remove, we include these perturbations as they597 offer a systematic approach to investigating model598 behavior, such as potential lead and recency biases. 599 Another potential solution to address robustness600 issues can be to use recent large language models to601 pre-process dialogues by removing errors and rep-602 etitions. However, this approach suffers from two603 challenges: (1) During deployment, additional pre-604 processing could increase latency, and (2) language605 models may hallucinate content, posing the risk of606 introducing factual errors in the input dialogue. 607 Finally, we examine if training with perturba-608 tions can help to mitigate robustness issues. We609 fine-tune BART on the training data augmented with610 perturbations and re-evaluate its performance. We611 create multiple training datasets, each modified by612 a specific kind of perturbation (typographical errors613 and language use variations for utterance level; rep-614 etitions, split utterances, and greetings for dialogue615 level), using TweetSum’s training split. These mod-616 ified datasets, with 5-50% of dialogues perturbed,617 are used to fine-tune BART, which we then test on a618 similarly altered TweetSum’s test split. 7 We hypoth-619 esize that training with more perturbed dialogues620
7We experimented with training and evaluating a single model on data with all perturbations. However, since different perturbations can have varied impacts on model performance, we found perturbation-wise analysis more interpretable. will initially improve performance until a threshold, 621 after which overfitting may reduce effectiveness. 622
Figure 3 shows the change in model consistency 623 when fine-tuned with perturbations. The lower the 624 change in consistency, the higher the model robust- 625 ness to the perturbations. One takeaway is that 626 different perturbations necessitate varying amounts 627 of perturbed examples in the training set to achieve 628 maximum performance gain. For example, typo- 629 graphical errors and language use variations yield 630 the largest drop in ∆zc when approximately 40% 631 and 45% of the dialogues are perturbed during train- 632 ing. In contrast, dialogue-level perturbations re- 633 quire significantly less perturbed data during train- 634 ing, with approximately 30% split-utterances, 15% 635 greetings, and only 5% repetitions being sufficient. 636 Overall, the results demonstrate that fine-tuning 637 with perturbed data does not yield consistent per- 638 formance improvements, warranting more detailed 639 exploration as part of future work