CorDial: Coarse-to-fine Abstractive Dialogue Summarization with Controllable Granularity
Authors: Chien-Sheng Wu
Linqing Liu
Wenhao Liu
Pontus Stenetorp
Caiming Xiong
Venue:  (2021)
Keywords: dialogue
summarization
controllable generation
natural language processing

Abstract: Dialogue summarization is challenging due to its multi-speaker standpoints, casual spoken language, and limited labeled data. In this paper, we propose CorDial, aiming to improve the abstractive dialogue summarization quality and at the same time enable granularity controllability. We propose 1) a coarse-to-fine generation strategy that generates a summary draft followed by a final summary in an autoregressive way. The summary draft, which provides weakly-supervised signals, is composed of pseudo-labeled interrogative pronoun categories and noisy key phrases extracted with a constituency parser. 2) A simple strategy to control the granularity of the final summary. CorDial can predict and control the number of summary sentences for a given dialogue by predicting and highlighting different text spans from the source text. Our model achieves state-of-the-art performance on the largest dialogue summarization corpus SAMSum. We conduct comprehensive error analysis and show competitive human evaluation results to annotated summaries.

Under review as a conference paper at ICLR 2021
CORDIAL: COARSE- ABSTRACTIVE DI-
ALOGUE SUMMARIZATION WITH CONTROLLABLE GRANULARITY
Anonymous authors Paper under double-bling review
ABSTRACT
Dialogue summarization is challenging due its multi-speaker standpoints, ca- sual spoken language style, and limited labeled data. In this paper, we propose CORDIAL, aiming to improve the abstractive dialogue summarization quality and, at the same time, enable granularity controllability. We propose 1) a coarse-to- fine generation strategy that generates a summary draft followed by a final sum- mary. The summary draft, which provides weakly-superviied signals, comprises pseudo-labeled interrogative pronoun categories and noisy key phrases extracted with constituency parser. 2) A simple strategy to control the granularity of the final summary. CORDIAL can automatically determine or control the number generated summary sentences for given dialogue by predicting and highlight- ing different text spans from the source text. Our model achieves state-of-the-art performance the largest dialogue summarization corpus SAMSum. We con- duct comprehensive case study and show competitive human evaluation results and controllability to human-annotated summaries.
INTRODUCTION
Text summarization tasks aim distill the most critical information in a text to produce an abridged version. In particular, abstractive erative models with high level of semantic understanding, as the output words not necessarily appear in the source text. It is more challenging but gives much flexibility to summary compared to any extractive summarization models (Zhang et al. 2018). Abstractive dialogue summarization has been discussed in the literature of AMI meeting corpus (McCowan al. 2005). The size and qual- ity of labeled data are bottlenecks, as collecting summaries is costly and judgements when creating them inherently subjective. The AMI corpus has only 141 summaries, and the largest dialogue summarization dataset SAMSum (Gliwa et al., (201)) has number of training samples only equal to 5% of the commonly-used text summarization dataset CNN/Daily (Hermann al. (015).
opposed to extractive summarization requires neural gen-
In addition to (and perhaps due to) the shortage of labeled data, dialogue summarization has not received much attention despite the prevalence of dialogues (text messages, emails, social media) and the vast application potential of dialogue summarization systems. Significant research efforts have been focused on summarization of single-speaker documents such News (Hermann al. 2015 Nallapati et al. 2016 See et al., 2017) or scientific publications (Qazvinian & Radev] 2008 Nikolov et al. 2018). However, summarizing a dialogue presents unique set of challenges. A conversation always involves multiple speakers with different points of view, and its natural language style very different from a standard writing style. For example, conversational data contains more abbreviations and typos. Information is more scattered across in a dialogue, compared to articles where usually titles or the first few sentences contain the most salient information.
Recently, the ability to control text summarization in the News domain has been gradually attracting more attention (Fan et al.) 2018 Liu et al.) 2018), with work focusing on learning length embeddings to control summary lengths. However, the length information is only added during the decoding stage, making the encoding stage less informed, and the overall conditional generation implicit. Saito et al. (2020) instead first explicitly extract "prototype" text span in the desired length and then

---

Under review as a conference paper at ICLR 2021
Morgan
ship Hey gorgeous, what's up?
Turn
Intent
Key Phrase
Suzanne
Nothing special, it's just one of many boring days at work. But's better now
what
abotain
"a just one of many boring days at work"
Morgan
Are you working at all
confirm
"working at all"
Suzanne
I'm trying but you aren't helping all. I'm
abstain
'm just taking well-deserved break
just taking a well-deserved break kihll
abstain
Morgan
Shiz miss you Suzle
confirm
"leel like
"is playing week", madison square gardin"
Suzanne
miss you too Morgan
Morgan
Do you feel like going concert next week? maroon playing at the hulu theater at madison square garden
why
happens I've got two tickets. do you want to go sthls
abstain
Suzanne
shis Really? OMG! That's wonderful Thank you sweetheart!
Summary: Suzanne is at work and having break now. Morgan invites Suzanne 5 which takes place next week the Hulu Theatre at Madison Square Garden. Suzzanne agrees.
Maroon
Morgan
Oh, nothing.
just want to be happy
Sihis
An input and output example for our proposed solution. Given the dialogue the left hand side, we first construct summary draft with intent and key phrase information for coarse-to-fine generation. Then, we split the dialogue into several pieces by special tokens for model controllability and interpretability.
Figure I:
paraphrase it to the output News summary. However, the retrieve-and-rewiid process is restricted by the extraction quality, leaving its performance limited by extractive solutions' capabilities.
In this paper, we propose CORDIAL, a coarse-to-fine abstractive dialogue summarization model equipped with granularity controllability. Unlike previous methods (Goo & Chen 2018: Pan et al, 2018) which heavily rely on explicit intent annotations in datasets, we label each dialogue turn with a pre-defined interrogative pronoun category using a weakly-supervise automatically labeled user intent together with its corresponding key phrase extraction provide weak supervision during summary generation. In addition, we propose length-comtrollabbe generation method specifically for dialogue summarizaiton. We match each summary sentence "linearly" to its corresponding dialogue context and clip it by highlighting tokens. We then train our model to predict where clip and generate only one sentence for each highlighted dialogue. This strategy enables CORDIAL to generate a summary at different granularity by highlighting arbitrary numbers of text spans from a dialogue and making our model more interpretable.
labeling approach. The
We build our model on top of BART-xsum (Lewis et al., 2019), which is first pre-trained with unsupervised denoising objectives, and further fine-tuned on the News summarization corpus XSUM (Narayan et al. 2018). In the experimental results, show that CORDIAL achieves state-of- the-art dialogue summarization performance on several automatic metrics. The main contributions of this work are: 1) We propose coarse-to-fine strategy that artificial summary draft weak supervision, 2) we introduce a text-span based conditional generation approach to control the gran- ularity of generated dialogue summaries without human- summaries at different detail levels, and 3) we conduct comprehensive case study and human evaluation to show that CORDIAL can achieve consistent and informative summary, especially for controllable summary, where existing models either cannot do it or do it poorly.
2 METHODOLOGY
In this section, we first briefly cover the background of generative language pre-trained models. Then, introduce proposed summary draft construction and summary controllability in detail. The proposed solution is generalizable to all the generative language models. We define the conver- sational history input D (X,, X2,..., Xx), where each X, has sequence of words, N is the
Our code is released at www. www.anonymous.com
2

---

Under review as a conference paper at ICLR 2021
total numbers of dialogue turns, and the input may contain more than two speakers. We intend to generate M -sentence dialogue summary Y {Yi,..., Fa....,Ya. Yas) that is suppose to be briefer than the overall dialogue history.
2.1
GENERATIVE PRE- TRAINED LANGUAGE MODELS
Self- supfrsupereieed generative language models (Radford et al. 2019 Yang et al. 2019 Dong et al, 2019 Lewis et al.] 2019) have achieved remarkable success many NLP tasks. Instead of solely traiing on a specific dataset, pre- vre-trainig on a large-scale text corpus such as Wikipedia and BookCorpus has shown good knowledge transferahility. However, in conversational AI, previous works (Wolf' et al. 2019b Wu et al. 2020) argue that there is an intrinsic difference of linguis- tic patterns between human conversations and writing text. We would like to answer the follow- ing question: which generative language model is the best base model for dialogue summarization tasks. In this paper, we investigate four recently proposed models, DialoGPT (Zhang et al. 201 d),,d), UniLM (Dong et al.] (2019), PEGASUS (Zhang et al., (019a), and BART (Lewis et al. [0199). The first one is a generative language model trained on open-domain conversational data, and the other three achieve promising results on several text summarization benchmarks.
2.2 DRAFT CONSTRUCTION
For each turn in the dialogue, we create summary draft containing the speaker's dialogue action category (defined later) together with the most critical key phrases. Our hypothesis is such summary drafts can provide useful weak supervision beneficial to the final summarization task. Unlike task- oriented dialogue systems, which have explicit and annotated intents (e-g., book flight and check account) and actions (c.g., inform and request), dialogue summarization tasks rarely have such la- bels. Therefore, we define a set of interrogative pronoun categories. and then label each utterance with its category using weakly-supervised labeling approach (Ratner et al. 2019
The interrogative pronoun categories are designed to identify functional units (intent) of all utter- ances, serving as the dialogue's s logic. For example, in Figure []] Morgan asked Suzanne "Do you feel like going to a concert next week?" One can expect that Suzanne will confirm her willingness in the next dialogue turn. We define such dialogue action categories reason of the state mentioned in the previous turn, e.g., "why" or "why not"; WHAT: requests more details about the aforemennioned object, the sentence usually starts with "what's" or "what about"; WHERE: asks the location of an appointment or event; WHEN: asks the time of an appointment or event, e.g. "when" or "what time": CONFIRM: asks the other speaker to establish the correctness of certain case, the sentence usually starts with patterns like "are you". "wwll you" or "has he"; AB- STAIN: the utterance does not belong to any of the previous categories. It occurs when speakers continue to state or comment without seeking for more information from the others.
follows: WHY: asks the
Following this, we define key phrases as essential words or phrases in the annotated summary. We first use trained constituency parser (Kitaev & Klein 2018) to parse each dialogue turn and each summary sentence. Then, we identify the longest common sub-sequence with a threshold among dialogue and summary to be the key phrases. Note that not every dialogue turn contains key phrases and the key phrases could be noisy. Overall, we construct a summary draft by concatenating turn index, dialogue action category label, and extracted key phrases within the entire dialogue history into a string, ending with a special token, "TL;DR". Take Figure an example, the summary draft is "I what 2 abstain 's just one of ... square garden why abstain TL;DR". We train our model first to generate this summary draft and then generate the final summary in an autoregressive way. We use TL:DR token to distinguish draft and final summary during inference time.
2.3 CONTROLLABILITY
The high-level intuition for our solution is that if we can control generative model only to gen- crate one sentence as output for a purtially-hgghlightee input, we can control the number of output sentences by choosing how to highlight the input. We highlight each dialogue split using the spe- cial token pair < hl > and < /hl >. For example, in Figure [ we generate the first summary sentence for the first segment marked in blue color with the highlighting token pair, and the second
i www.tuu.cmmsnncnoo wteerrkeeeeeeeeeeeeee -ettteeamsseeee...... a
3

---

Under review as a conference paper at ICLR 2021
and third by green and orange markers, respectively. This way, not only gain the summary controllability but also make the generation more interpretable.
The next challenge is, during training, have to find a mapping between each sentence in refer- ence summary to its corresponding dialogue split. In other words, how do we know where to insert the highlighting tokens? We do by training a dialogue-turn- turn-level binary classifier (detailed below) that predicts whether each turn is cutting point. Our key observation is that sentences within a ref- erence summary usually have a strong temporal dependency, that is, people summarize the dialogue almost "linearly". We use a naive approach that simply find the dialogue split that has the highest ROUGE score to each summary sentence. The cutting point
Im arg max; SIM(X Ym))
(1)
where SIM could be any similarity functions (we use ROUGE-I) available, and Cm the accumu- lated turn index (c1 = 1 and Cm 1m-1) that indicates which part of dialogue has been covered by a summary sentence. fm is the cutting point in the dialogue history for the m-th summary sen- tence. Note that for a summary with M sentences, we only need M - 1 cutting points. With the pseudo labels (lm) provided by this heuristic, we formulate the dialogue segmentation problem into binary classification problem. Specifically, we train classifier C. which takes dialogue history as input and predicts whether each dialogue turn is a cutting point. We prefix each dialogue turn with separation token (e.g., Fsep =< >) and take such long sequence as input to the classifier.
H aep: X1, Tseps X2..... Ssepp. XNl) Nxdemb,
p = (H)) E ER.
The classifier output H is the representations of those separation tokens, and each of them is demb dimension vector. W1 E Rdemss x1 is a trainable linear mapping. The P is the predicted segment probability that is trained with binary cross-entropy loss. We use a BERT-base model (Devlin al,) 2018) as classifier C' and the i-th cutting point triggered P; > 0.5. This prediction means that our model can automatically determine how many sentences should be generated in the final summary. If no cutting point is triggered, then we will generate a one-sentence summary. If one cutting point is triggered, we will have two-sentence summary, and on. Note that the segment classifier does not need to be perfect (e.g.. 86% accuracy in our experiment) as the labels contain certain noise and splitting dialogue by one turn earlier or later usually does not make big difference. As a result, we find that the final ROUGE score is quite similar for both "oracle" dialogue splits and predicted dialogue splits.
Finally, we can control the number of output summary sentences by controlling the dialogue split. Specifically, we first decide the expected number of output sentences (e.g., K), and then we choose the top K- indexes with highest probabilities in segmentation probability P. We use these K - indexes as cutting points to add highlighting tokens. For example, if we want to have a three- sentence summarization, we can split the dialogue into three pieces by choosing the top two highest segment probability. Then, we input the same dialogue three times with different highlighting por- tions to CORDIAL and obtain one-sentence summary each. We can also generate one-sentence summary by clipping the whole dialogue with one pair of highlighting tokens at the beginning and the end of a dialogue (we call this output CORDIAL-I).
2.4 OVERALL GENERATION
CORDIAL follows standard encoder-decoder framework. During training, we use "oracle" dia- logue segmentation to add highlighting tokens for each summary sentence, separately. We take the highlighted dialogue history as input and train our model generate its corresponding summary draft and a summary sentence. For example, the first summary sentence, we input whole dialogue with added highlighting tokens both in the beginning of the first turn and in the end of the fourth turn, and generate output that contains corresponding summary draft "1 what 2 abstain ... well-deserved break" and the first summary sentence "Suzanne is at work and is having a break now." The entire model is trained on cross-entropy loss for the generated tokens. During inference, we first use the trained segment classifier to predict splitting points, suggesting how many sentences to be generated in the final summary. Then, we use the predicted segmentation add highlighting tokens into dialogue. Finally, after generating multiple summary sentences separately, we simply concatenate them to be the final summary.
4

---

Under review as a conference paper at ICLR 2021
3 EXPERIMENTS
3.1 DATASET
We perform experiments on the recently released SAMSum dataset (Gliwa et al. 2019), which to the best of our knowledge the most comprehensive resource for abstractive dialogue summarization tasks. It contains 16K natural messenger-like dialogues created by linguists fluent in English with manually annotated summaries. Its comprehensiveness is reflected in the following aspects: 1) Unlike previous datasets consisting of only hundreds of dialogue- dialogue-summary pairs, it has larger data size 14,732 training pairs, 818 validation pairs, and 819 test pairs; 2) 75% of the conversations are between two interlocutors, the rest are between three or more people; 3) the conversations cover diverse real-life topics, and the summaries are annotated with information about the speakers.
We preprocess the data by the following steps: 1) concatenate adjacent utterances of the same speaker into one utterance; 2) clean the dialogue text by removing hashtags, URLs and Emojis; 3) label each utterance with its corresponding interrogative pronoun category with weak supervi- sion approach (Ratner et al. 2019;; 4) parse each utterance with constituency parser and find the longest common sub-sequence between the phrases and summary to be the key phrases.
3.2 EVALUATION METRICS AND BASELINES
We the standard ROUGE metric (Lin 2004) automatic evaluation metrics, including ROUGE- 1. ROUGE-2, and ROUGE-L. Following previous works (Gliwa et al. 2019), we use RY-ROUGE library with stemming function. We compare our model with baselines reported in Gliwa et al. 2019) Longest-3 is commonly-used extractive summarization baseline which takes the top three longest sentences as summary. The pointer generator and Fast abs are RNN-based methods with copy-attention mechanism policy gradient. The Transformer is random-initialized self-attention architecture with multi-head attention. The DynamicConv is lightweight convolutional model that can perform competitively to self-attention. All of these models are not pre-trained.
Besides, we investigate four pre-trained generative language models. DialoGPT is GPT model pre- trained on open-domain Reddit data. UniLM is pre-trained using three types of language modeling tasks: unidirectional, bidirectional, and sequence-to-sequenee prediction on English Wikipedia and BookCorpus. PEGASUS masks important sentences from input and is trained to generate the miss- ing parts, similar to an extractive summary approach. BART is trained by corrupting text with an arbitrary noising function and learning to reconstruct the original text. We use default parameters listed in the respective open-source repositories to fine-tune on the dialogue summarization task. We show the training details in the Appendix.
3.3 RESULTS
In Table of ROUGE results, we find that the methods that are pre-trained or with pre-trained embeddings perform better than those that are not. For instance, DynamicConv achieves a 3-4% improvement by adding GPT-2 embeddings. This further confirms the impact of language model pre-training downstream tasks. Among the pre-trained generative language models examined, PEGASUS and BART are the top two models with ROUGE-1 higher than 50. Surprisingly, Di- aloGPT, the model pre-trained conversational data, does not achieve satisfactory results one might expect.
CORDIAL achieves the highest 50.79% ROUGE-L score. We also conduct ablation study by moving summary draft generation (BART+Cth) or controllability (BART+Draft). In both cases we observe a performance drop, except slight improvement on ROUGE-1 for BART+Cul. This sug- gests that the drafting step helps generate a more fluent summary even with lower unigram matching. Furthermore, recognizing the limitation of ROUGE scores in their ability to fully capture the resem- blance between the generated summary and the reference, in Table we follow Fabbri et al. (2020) to compare model performances with additional metries, including ROUGE- Word Embedding (Ng & Abrecht 2015), BERTScore (Zhang et al.] 2019b), MoverScore (Zhao et al. 2019), Sentence Mover's Similarity (SMS) (Clark et al. 2019
BLEU Papineni et al.) 2002), and CIDFr (Vedantam
pypi pypi.orr//project//ee
5

---

Under review as a conference paper at ICLR 2021
Table 1:
Dialogue summarization ROUGE evaluation on the SAMSum test dataset. Results with are obtained from Gliwa et al. (2019). CORDIAL achieves the highest ROUGE score.
ROUGE-1
ROUGE-2
ROUGE-L
Longest-3
32,46
10.27
29.92
Pointer Generator (See 2017
37.27
14.42
34.36
Fast Abs RL (Chen & Bansal
2018
41.03
16.93
39.05
Transformer Vaswani
2017
42.37
18.44
39.27
DynamicCon
2019
41.07
17.11
37.27
DynamicComy + GPT - emb
45.41
20.65
41.45
DialoGPT (Zang al.
2019d
39.77
16.58
38.42
UniLM Dong et al.
2019
47.85
24.23
46.67
PEGASUS (Zhag ct al.
2019a
50.50
27.23
49.32
BART
Lewis et
2019)
51.74
26.46
48.72
BART
Draft
51.79
26.85
49.15
BART + Curl
52.84
27.35
50.29
CORDIAL 52.65
27.84
50.79
Table 2: Dialogue summarization evaluation on the SAMSum test dataset with additional recently introduced metries that have been applied to both text generation and summarization.
ROUGE_ WE
BERTScore
MoverScore
BLEU
CIDEr
SMS
PEGASUS BART
0.3562
0.5335
0.3233
17.33
1.741
0.1608
0.3606
0.5387
0.3391
17.55
1.701
0.1401
CORDIAL 0.3759
0.5458
0.3539
19.58
1.981
0.1689
et al. 2015). As shown in T CORDIAL consistently outperforms PEGASUS and BART. More information about these evaluation metrics are shown in the Appendix.
3.4 ANALYSIS
3.4.1
HUMAN EVALUATION BY CROWDSOURCING
We leverage human judgement to evaluate the generated summaries via crowdsourcing, especially generation, since we not have numan-written reference summaries of various lengths (number of sentences). We ask workers to rate the summaries in two aspects scale from -1 (worst) to (best): factual consistency and informativeness. Factual consistency acts as a precision measure, assessing whether the information provided in summary contains factual errors. Informativeness is a recall-oriented measure, examining whether critical information a di- alogue is mentioned in summary. We also show the length ratio between summary and a dialogue, where a lower ratio means a higher compression rate. For the crowsourcing evaluation, we randomly select 30 dialogues, each of which is annotated by three workers.
for granularity-coniee
To show the proposed controllable generation's 's strengthens and quality, we provide two additional baselines, Longest-1 and BART-1 The longest-1 method is extractive baseline that outputs the longest dialogue turn as the final summary. The BART-1 is a strong abstractive baseline where we train BART-based summarization model with the number of summary sentences in the training as its can use different -farteef--entence tokens control the BART output.
token during decoding. Similar to the approach from Liu et al. (2018), we
In general, it is preferable to have factually consistent and informative summary that is succinct (low length ratio, high compression rate) the same time. As shown in the first row of Table 3 CORDIAL-1 achieves the highest informative score among all generated one-sentence summaries, indicating the strength of proposed controllable method producing suceinet yet informative dialogue summaries. The Longest-1 method has higher consistent score because its summary is directly copied from the original dialogue, preventing any factual mistakes. The second row of Table shows that CORDIAL, when automatically determining the granularity of the summary, produces summaries that are more succinet (lower length ratio), more factually consistent, and more informative, compared to the BART model.
*The prediction file the set is provided in the supplementary file.

---

Under review as a conference paper at ICLR 2021
Table
Human evaluation results on test set for both controllable summary and standard summary.
Length Rabo
Consistent (Precision)
Informative (Recall)
Longest-
0.27
0.70
0.23
BART-1
0,16
0.50
0.16
CORDIAL-1
0.19
0.50
0.49
BART
0.26
0.65
CORDIAL
0.24
0.69
0,53
Gold
0.27
0.74
0.55
Table 4: A test set example with generated summaries.
Kelly: I still haven't I received the rent money. Did you check with your bank?
John: Yes. definitely
last week.
Kelly: But I still don't have it. Can you plense check that you sent it to the right account.
John: Ok. Give me min.
Kelly: OK
John: I checked and the money did go out of my account last week.
Kelly: What account number did you send it to?
John: 44-1278
Kelly: No wonder! My account number is 44-1279. You sent it to someone else's account.
John: Fick! D*mm! Frek!. I'm really sony!
Kelly: still need the money though.
John: I'm really somr I'll have to to the bank tomorrow and ask if they can re-send it to the right account.
Kelly: Thanks
Longest-
John I'm really sorry have go the bank fomorrow
they can re-send to the right account.
BARI-1
Kelly still hasn't It received the rent money from John.
John scat the rent money to the wrong account and will have to ask the bank to re-sead it to the correct one tomorrow.
CORDIAL-]
the wrong account number 44-1278. John will go to the bank tomorrow and ask if they can re se-send the money to the right account.
Kelly still hasn't received the rent money. John
BART
John sent the rent money to the wrong account last week. John will go to the bank tomorrow and ask he can re-send
CORDIAL
money to the Kelly hasn't I received the rent money, because John sent it to the wrong bank account. He will go the bank to tackle the issue.
account.
Gold
3.4.2 CASE STUDY
CORDIAL outperforms the baseline models in both ROUGE scores and human evaluation metrics. We now further inspect its textual quality. In Table 4 we show an example from the SAMSum test set with summaries generated by different models. In this example, CORDIAL and CORDIAL-1 can both produce a near-perfect summary even compared to the human-written reference summary. On the other hand, the summary generated by BART includes overly detailed information (e.g., bank account). We show some more examples in the Appendix and all the predictions (including CORDIAL-1 and CORDIAL-2) in the supplementary file.
We also manually examine 100 summaries generated from CORDIAL against the reference sum- maries in the test set. Specifically, we analyze each of the three following problematic cases, where summarization models frequently make mistakes, reported by Gliwa et al. (2019) and provide sam- ple summaries in Table 1) Associating names with actions: CORDIAL performs well in dealing with speakers' names. It accurately associates "her dad' with "Ann's dad," also "Fiona tries to help her" with "Ann and Fiona." 2) Extract information about the arrangement after discussion: Even speakers hesitate about the flower's ' color to be yellow, pink or red in the middle of the discussion, CORDIAL still correctly determines the right color after several turns. 3) Decide important informa- tion in dialogues: CORDIAL fails to capture some of the important facts (marked as red) mentioned in reference summary. We conjecture the reason could be that 1) some of the important facts are located in the same part of the highlighted turns, and 2) those information is missed by the key phrase extraction. Simultaneously, we force the model to generate only the most important one un- der the constraint of controllability. The improvement of CORDIAL on the first two summarization difficulties can be partially attributed to the clear logic in the draft when an input to the model.
RELATED WORK
Neural Text Summarization There are two main paradigms for text summarization: extractive and abstractive. Inspired by the success of applying seq2seq models on neural machine translation,
7

---

Under review as a conference paper at ICLR 2021
Table 5: Case analyses by manually examining CORDIAL generated summaries.
Reference Summary
CORDIAL Summary
Associat actions
will he late.
Lally will for the meeting with Gihrriel.
Gabriel will order pasta salmon and basil her.
Gabriel order something for Lilly.
Ann doesn Know what she should give to hee dnd is a biriddyy gii..
It's Ann s dadlss 30th hirihdayy
He's turning 50.
He's turning 50. Aun and Fiona planning
Fiona help
suggests a painthall match.
surprise birtbday pany for her
Extract information after the discussion
Paul will buy red roses follewing Cindy's 's advice.
Paul wants to buy red roses.
Dacide important information
Rachel's aunt an accident
hospital now.
Rachel
hospital her aunt. who an accident.
She's only bruised.
The perpetrator of the accident going to for the rehabilitation.
Shi's bruised but line.
She will give her hug.
needs Betty's number She needs to contact Lary.
doesn't have
Amanda
Berty's number. Amanda suggests to text him.
Rush et (2015) and Nallapati al. (2016) introduce the neural seq2seq model on abstractive text summarization, with an attention- based encoder and a neural language model decoder. To solve the problem of out-of-vocabule vocaalloocaallary words and capture salient information in source documents, See et al.) (2017) propose Seinter-generator network that copy words from source to target. Many subsequent works (Gehrmann et al., 2018 Paulus et al. 2018) employ this mechanism and further demonstrate its effectiveness with reinforcement learning. Recently, Liu & Lapata (2019) apply BERT on text summarization and propose a general framework for both extractive and abstractive models. Zhang et al. (2019c) pre-train hierarchical document encoder for extractive summarization. Lewis et al. (2099) introduces BART, a denoising autoencoder for pretraining sequence-to-sed -esuueeec models. BART significantly outperforms the best previous work in terms of ROUGE metrics.
Dialogue Summarization Most research works (Oya et al. 2014; Mehdad et al. 2014 Banerjee et al. 2015) are conducted on the AMI meeting corpus (MCCowan et al. 2005), Goo Chen (2018) propose to use the topic descriptions as reference summaries and use dialogue acts as clear training signals. However, such summaries are very general since they only describe the high-level goals of meetings. Pan et al. a (2018) build the Dial2Desc dataset by reversing visual dialogue task. [iu et al. (2199) collect their dataset from the logs in the DiDi customer service center. However, the dataset not publicly available. It is restricted to the task-oriented scenario, where one speaker is the user and the other is the customer agent, with limited topics. Recently, Gliwa et al.) (2019) introduce the SAMSum corpus, which contains around 16k chat dialogues with manually annotated summaries. It is the first comprehensive abstractive dialogue summarization dataset spanning over various lengths and topics.
Length-controllabh Generation The most prevalent method for length control generation is us- ing a special length embedding. Kikuchi et al. (2016) first propose length control for abstractive summarization by using length embedding train embeddings that correspond to each different output length and prepend that unique length marker at the beginning of the decoder. Liu et al. (2018) incorporates the length embedding into the initial state of a CNN-based decoder. Takase & Okazaki (2019) extends the positional encod- ing in Transformer model by considering the remaining length explicitly each decoding step. Saito et al. (2020) propose to control the summary length with prototype extractor, but the sum- mary rewriting process is largely bounded the extraction quality. The aforementioned works all focus on structured text summarization(c.2 news document). We are the first to propose generate length-controllable summary on dialogues by highlighting arbitrary numbers of dialogue spans.
additional input for the LSTM. Fan al. (2018)
CONCLUSION
The dialogue summarization task is challenging but with vast application potential. We propose CORDIAL, a state-of-the-art dialogue summarization model with granularity controllability. COR- DIAL uses a weakly-labeled summary draft for its coarse-to-fine generation, and text-span condi- tional generation for a controllable summary. Our model achieves state-of-the-art art results on the largest dialogue summarization dataset. We show with human evaluation that our model can gen- erate factually consistent and informative summaries. We also point out several error cases to shed light on future research direction of controllable dialogue summarization.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Siddhartha Banerjee, Prasenjit Mitra, and Kazunari Sugiyama. Abstractive meeting summarization using dependency graph fusion. In Proceedings of the 24th International Conference on World Wide Web, pp. 5-6, 2015.
Yen-Chun Chen and Mohit Bansal. Fast abstractive summarization with reinforce-selected sentence rewriting. arXiv preprint arXiv: 10080, 2018.
Elizabeth Clark, Asli Celikyilmaz. and Noah Smith. Sentence mover's ' similarity: Automatic evaluation for multi-sentence texts. In Proceedings of the 57th Anmual Meeting of the Association for Computational Linguistics, pp. 2748-2760, 2019.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.
Li Dong, Nan Yang. Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao-Wuen Hon. Unified language model pre-training for natural language understanding and generation. In Advances Neural Information Processing Systems, pp. 13063-13075, 2019.
Alexander Fabbri, Wojciech Kryšciñski,, Bryan McCann, Caiming Xiong, Richard Socher, and Dragomir Radev. Summeval: Re-evaluating summarization evaluation. arXiv preprint
arXiv:2007.16666, 2020.
Angela Fan, David Grangier, and Michael Auli. Controllable abstractive summarization. In Pro- ceedings of the 2nd Workshop on Neural Machine Translation and Generation, pp. 45 45-54, 2018.
Sebastian Gehrmann, Yuntian Deng, and Alexander Rush. Bottom-up abstractive summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 4098-4109, 2018.
Bogdan Gliwa, Iwona Mochol, Maciej Biesek, and Aleksander Wawer. Samsum corpus: A human- annotated dialogue dataset for abstractive summarization. arXiv preprint arXiv: 19X1v:99111122511 2019.
Chih-War Goo and Yun-Nung Chen. Abstractive dialogue summarization with sentence-gand mod- cling optimized by dialogue acts. 2018 IEEE Spoken Language Technology Workshop (SLT), pp. 735-742. IEEE, 2018
Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. Teaching machines to read and comprehend. In Advances in neural information processing systems, pp. 1693-1701, 2015.
Yuta Kikuchi, Graham Neubig, Ryohei Sasano, Hiroya Takamura, and Manabu Okumura. Con- trolling output length in neural encoder-decoders. In Proceedings of the 2016 Conference on Empirical Methods Natural Language Processing, pp. 1328-1188 1338, 2016.
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint 1rXiv:1412.6980 2014.
Nikita Kitaev and Dan Klein. Constituency parsing with self-attentive encoder. arXiv preprint arXiv:/805.0003: 2018.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy. Ves Stoyanov, and Luke Zettlemoyer. Bart: Denoising sequence-to-sequeeee pre- training for natural language generation, translation, and comprehension. arXiv preprint 2019.
Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pp. 74-81, 2004.
Chunyi Liu, Peng Wang. Jiang Xu, Zang Li, and Jieping Ye. Automatic dialogue summary genera- tion for customer service. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 1957-1965, 2019.

---

Under review as a conference paper at ICLR 2021
Yang Liu and Mirella Lapata. Text summarization with pretrained encoders. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th Inter- national Joint Conference on Natural Language Processing (EMNLP-IJCNLP),, pp. 3721-3731, 2019.
Yizhu Liu, Zhiyi Luo, and Kenny Zhu. Controlling length in abstractive summarization using convolutional neural network. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 4110-4119, 2018.
L. McCowan, J. Carletta, W. Kraaij, S. Ashby, S. Bourban, M. Flynn, M. Guillemot, T. Hain, J. Kadlec, V. Karaiskos, M. Kronenthal, G. Lathoud, M. Lincoln, A. Lisowska, W. Post, Dennis Reidsma, and P. Wellner. The ami meeting corpus. In L.P.J.. Noldus, F Grieco, L.W.S. Loijens, and P.H. Zimmerman (eds.). Proceedings of Measuring Behavior 2005, Sth International Con- ference on Methods and Techniques in Behavioral Research, pp. 137-140. Noldus Information Technology, 8 2005. ISBN 90-74821-71-5.
Yashar Mehdad, Giuseppe Carenini, and Raymond Ng. Abstractive summarization of spoken and written conversations based on phrasal queries. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 1220-1233 2014.
Ramesh Nallapati, Bowen Zhou, Caglar Gulcehre, Bing Xiang, et al. Abstractive text summarization using sqquence-to-squeene rmns and beyond. arXiv preprint arXiv:1602.0602.. 2016.
Shashi Narayan, Shay B Cohen, and Mirella Lapata. Don't give me the details, just the sum- mary! topic-aware convolutional neural networks for extreme summarization. arXiv preprint S.fv:1808.087444 2018.
Jun-Ping Ng and Viktoria Abrecht. Better summarization evaluation with word embeddings for ROUGE. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp. 1925-1930, Lisbon. Portugal, September 2015. Association for Computational Linguistics. doi: 10.18653/v/DD5--122 D15-1222
htRtttt:::/// /wwwwwwwww.///././/////. ac web. owt.bbbhoo//oo/ooooooooooo /
Nikola Nikolov, Michael Pfeiffer, and Richard HR Hahnloser. Data-driven summarization sci- entific articles. arXiv preprint arXiv::804 1804.08875 2018.
Tatsuro Oya, Yashar Mehdad, Giuseppe Carenini, and Raymond Ng. A template-based abstractive meeting summarization: Leveraging summary and source text relationships. In Proceedings of the 8th International Natural Language Generation Conference (INLG), pp. 45-53, 2014.
Haojie Pan, Junpei Zhou, Zhou Zhao, Yan Liu, Deng Cai, and Min Yang. Dial2dese: end-to-end dialogue description generation. arXiv preprint arXiv:1811.00188. 2018.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association forr Computational Linguistics, pp. 311-318, 2002.
Romain Paulus, Caiming Xiong, and Richard Socher. A deep reinforced model for abstractive summarization. In International Conference on Learning Representations, 2018.
Vahed Qazvinian and Dragomir R Radev. Scientific paper summarization using eitation summary networks. arXiv preprint arXiv:0807. 2008.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei. and llya Sutskever. Language models are unsupervised multitask learners. 2019.
Alexander Ratner, Braden Hancock, Jared Dunnmon, Frederic Sala, Shreyash Pandey, and Christo- pher Ré. Training complex models with multi-task weak supervision. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 4763-4771, 2019.
Alexander M Rush. Sumit Chopra, and Jason Weston. A neural attention model for abstractive sentence summarization. Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp. 379-389, 2015.
10

---

Under review as a conference paper at ICLR 2021
Itsumi Saito, Kyosuke Nishida, Kosuke Nishida, Atsushi Otsuka, Hisako Asano, Junji Tomita, Hi- royuki Shindo, and Yuji Matsumoto. Length-controllable abstractive summarization by guiding with summary prototype. arXiv preprint arXivv 2020.
Abigail See, Peter J Liu, and Christopher Manning. Get the point: Summarization with pointer- generator networks. arXiv preprint arXiv: 1oxii:/704..0 170404438, 2017.
Sho Takase and Naoaki Okazaki. Positional encoding control output sequence length. In Proceed- ings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pp. 3999-4004, 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, pp. 5998-6008, 2017.
Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. Cider: Consensus- bassed image description evaluation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4566-4575, 2015.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Transformers: satte-of-the-eee natural language processing. arXiv preprint arXiv: 2019a.
Thomas Wolf, Victor Sanh, Julien Chaumond, and Clement Delangue. Transfertransfo: A transfer learning approach for neural network based conversational agents. arXiv preprint arXiv:1901.08149, 2019b.
Chien-Sheng Wu, Steven Hoi, Richard Socher, and Caiming Xiong. Tod-bert: Pre-trained natural language understanding for task-oriented dialogues. arXiv preprint arXiv:2004.06871, 2020.
Felix Wu, Angela Fan, Alexei Baevski, Yann N Dauphin, and Michael Auli. Pay less attention with lightweight and dynamic convolutions. arXiv preprint arXiv:1901.10433 2019.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. XInet: Generalized autoregressive pretraining for language understanding. Advances in neural information processing systems, pp. 5753-5763, 2019.
Jingqing Zhang. Yao Zhao, Mohammad Salch, and Peter J Liu. Pegasus: Pre-training with extracted gap-sentences for abstractive summarization. arXiv preprint 1rXiv:19/2.08777, 2019a.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. Bertscore: Evaluat- ing text generation with bert. arXiv preprint arXiv: 1eXiv:1904.0 1e....9049999 09675, 2019b.
Xingxing Zhang, Mirella Lapata, Furu Wei, and Ming Zhou. Neural latent extractive document summarization. arXiv preprint arXiv: 1808.088..110 2018
Xingxing Zhang, Furu Wei, and Ming Zhou. Hibert: Document level pre-training of hierarchi- cal bidirectional transformers for document summarization. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 5059-5069, 2019c.
Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing Liu, and Bill Dolan. Dialogpt: Large-scale generative pre-training for conversational response generation. arXiv preprint arXiv:1911.0055. 2019d.
Wei Zhao, Maxime Peyrard, Fei Liu, Yang Gao, Christian M Meyer, and Steffen Eger. Mover- score: Text generation evaluating with contextualized embeddings and earth mover distance. 2019.
arXiv preprint
11

---

Under review as a conference paper at ICLR 2021
A
APPENDIX
A.1
DialoGPT: hhalopoh....h....commmooooo////d
UniLM: / htt.gggg...ooo//miccooootttttuu.
PEGASUS: http:h/hhhh.oooooooooeeaaaaaaaaa
BART htg.g html
A.2 TRAINING DETAILS
We use huggingface (Wolf al. 2019a) implementation to fine-tune BART model. We use the large version fine-tuned on the XSUM (Narayan et al. 2018) dataset with 12 self-attention encoder and decoder layers. We truncate input dialogue to maximal length 512 with training batch size 4. We train the model with Adam optimizer (Kingma & Ba 2014) with 0.1 proportion for linear learning rate warmup. We early stop on validation set ROUGE-I score, and it is trained for around 40,000 steps one NVIDIA V100 GPU. During inference, do beam search decoding with beam size
A.3 EVALUATION METRICS
Information obtains from Fabbri et al. (2020):
ROUGE measures the number of overlapping textual units between the generated summary and a set reference summaries.
ROUGE-W WE extends ROUGE by taking cosine similarity of Word2Vec embeddings into account.
BERTScore computes similarity scores by aligning generated and reference summaries on token-level based on the output of the BERT-basee model. Token alignments are computed greedily with the objective of maximizing the cosine similarity between contextualized token embeddings.
report the Fl score.
MoverScore measures semantic distance between a summary and reference text by making use of the Word Mover's Distance operating over n-gram embeddings pooled from BERT representations.
Sentence Mover's Similarity (SMS) extends Word Mover's Distance to view documents a bag of sentence embeddings as well as a variation which represents documents as both a bag of sentences and bag of words.
BLEU is corpus-level precision-foeuee a candidate and reference utterance and includes a brevity penalty. It is the primary evalu- ation metric for machine translation.
metric which calculates n-gram overlap between
CIDEr computes 1-4-gram co-occurrences between the candidate and reference texts, down-weighting common n-grams and calculating cosine similarity between the ngrams of the candidate and reference texts.
12

---

Under review as a conference paper at ICLR 2021
Table Test set example for qualitative study.
Phal good evening deuna ! many thanks for this nice card from you consiamiine was very happy are these sunglasses also Iroen
thought they belonged your
Phil: nope. she says they aren't hers.
Deana: mine either. look, mayhe you feel like keeping them
to many sunglasses...
Phil: where did you lind them, possible that they helong to adrian
Deana: this empply place above the radio in the very back adrian
exactly
Phil: ok. they don't belong to any us, and nobody clse drove your car. but we can look after these sunglasses.
Deana: glad hear it
Phil said good evening deana many thanks for this nice card from you. constantine was very happy
Longest-1
these sunglasses also from you ?
BART-1
Phil and Deana will look after Adrian's sunglasses.
CORDIAL-1
Deana found Adrian's sunglasses the back of Phal's
ВART
Phil and Deuna are going to look after Adrian's s sunglasses.
Carlial
from Deuna. Deana found them the empty place above radio. Deana has lot them. Phil recoived a card from Dsana. Coustantine was happy. Phil has sunglasses, that Deana found in the back above the radio.
Phil
Gold
Deann and Phil don't know who they helong Phil will keep the sunglasses.
Table 7: Test set example for qualitative study.
Colla: do you go holiday
Mike: i was thinking about egypt
Colia: too hot. what about
Mike: good idea i've never been there
Longest-1
Colia said where do you want go for holiday
BAKT-1
Mike wants to for holiday to Egypt.
CORDIAL-
Mike wants to on holiday to Egypt or Croatin.
BART
Celia Mike will go for holiday Crostia.
CorDial
Mike wants to
holiday Egypt. Colin thinks it's too hot, Mike has never been to Croatia, but he likes the idea.
Mike considers going to Egypt for holiday. It's too hot for Celia, she suggests Croutia instead. Mark likes the idea, he's been there.
Gold
Table 8: Test set example for qualitative study.
Diane: how long do you have to work tonight? ?
Ross: about hours why ?
Diane: just wanted to du something maybe
Ross: think i'll be worn out after all hat work, baby
Diane: we can just chill at home, don't worry. i just wanted to prepare
Ross: ok, then just to he safe let's say it will take me hours
Diane: but you just said
Ross: d Diane don't again
starting you'te impossible Diane: what
Ross: can't you understand that this is important me my carcer depends it
Diane: well.. your carcer is the most important thing in the world then wouldn't 't want to disturb !
Longest-1
Diane said well. if your career is the most important thing in the world then wouldn't 't want to disturb
BART-1
Ross has to work for hours tonight.
CORDIALL
Russ has to work hours lonight.
BART
Ross has work tonight hours. Ross and Diane will chill home.
CorDial
Ross has work hours tonight.
Gold
Diane not happy with Ross prioritising work over spending time with her.
13

---

Under review as a conference paper at ICLR 2021
Table Test set example for qualitative study.
Finn: bey
Zadie: hi there I what's up?
Finn: fine. you
Zadie: not bad. thanks
Finn: look, was thinking of going this neighborhood called elephant and castle tomorrow it's appurently full of
latin american stuff. fancy joining?
Zadie: but what's "tuff"
Finn: lol so apparently it's place were random people from latin america "( meaning flcck knows which countries) started running small businesses and restaurant and nice little community was formed
Zadie: oh cool
Finn: then capitalism came and it's all going to he demolished soon, so it's like the last chance to
Zadie: what shame yeah. haven't had latin american food ages I'm totally up for it
this cuisine unspecified latino origin lol. but can specify time and place and only you wish Zadie: might be tempted to lol i'd say early evening 2 ish?
Finn: wait
Finn: yeah. that's fine by me so most of the places we want to visit are in elephant and castle shopping centre see
the main entrance, wherever that
Zadie: 2 o' c'clock at unspecified main entrance then sounds good to mw
Finn: yer
Zadie: cool, you there and thanks much remembering about
Finn: thanks for saying yes to such an ill-defined plan Imao
Zadie: you know love those
Finn: see you tomorrow then
Zadie: yep call me if you get lost
Finn: will bye
Zadie: toodles
Pinn said yeah, that's s tine by me. most of the places we want to visit are in this elephant and castle
Longest-1
shopping centre, shall see you main entrance. wherever that
BAKI-
Finn and Zadie will tomorrow o clock the Elephant Castle shopping centre.
CORDIAL-1
Finn and Zadie going the Elephant and Castle tomorrow.
Finn and Zadie are going to eat Latin American food tomorrow. They are meeting at the Elephant and
BART
Castle shopping centre at pm.
DirDial
Latin Americun neighbourhood lomorrow, Zadie will call Finn he gets lost. Pinn and Zadie are going Elephant and Castle tomorrow 2. They will meet at the main entrance.
Pinn and Zadie going
Gold
Table 10:
Dialogue for the "Extract information after the discussion" sample in Table
Paul: what color flowers should get
Cindy: any just not yellow
Paul: ok pink? ?
Cindy: no maybe red
Paul: just tell me what color and what type ok ?
Cindy: ugh red roses
Gold
Paul will buy red roses following Cindy's ' advice.
BART
Paul wants to get red roses. Cindy doesn't 't want pink yellow roses.
CorDial
Paul wants to buy red roses.
14


Reviews:
[{"review_id": "lvs6FUDF8lj", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper is well written and investigates dialogue summarization that has not received much attention. It proposes a new model called CORDIAL which can generate a summary draft followed by a final summary.  It achieves comparable or better results in term of both automatic evaluation metrics, e.g. compress ratio, rouge score, and human evaluations (Consistent and Informative) about the quality of generated summaries in different settings. \n\nHowever, there are still several disadvantages of this paper:\n(1) It can generate a summary draft but its quality is almost not presented in the paper except the ablation study of table 1. Even the results within table 1 are still about the quality of the final summary. The paper slightly overclaims its usefulness in the draft summary generation. More results or analysis about draft summary should be presented.\n(2) The human evaluation has only 30 examples and the scale is too small. Also, does the score is -1, 0, 1, or other scale? Do you use majority vote or mean plus standard deviation to get results in the table? Why gold in table 3 is so low in Consistent?\n(3) The method looks applicable to the generable summarization task. More results of this are also interesting. \n\nAlthough these drawbacks, its quality is good overall.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "QOnlVerBmC1", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes CorDial, a new method for dialog summarization. CorDial firstly constructs a coarse draft by generating intent and key phrases for every dialog turn, and splits the dialog into chunks by inserting special boundary tokens; then the segmented original dialog and the constructed draft are feed as input to generate the final summary. CorDial employs BART-xsum as its backbone model, which is a pre-trained language model finetuned on XSUM summary dataset. Experiment result on SAMsum dataset shows CorDial achieves SotA performance under both automatic evaluation metric and human evaluations.\n\nOverall, the paper presents an interesting, practical recipe for dialog summarization. It requires few additional annotation besides the summary, and the human evaluation results looks promising. However, the method proposed here somewhat lacks in novelty, and some part of the paper is not clearly written. Thus I give this paper a weak reject rating.\n\nComments:\n1. In 2.2, how are the intents annotated? Is it a purely automatic process based on keywords matching? Or the keywords are merely cues for human annotators?\n2. In 2.3, the algorithm for finding the cutting points is an incremental one. It can't account for the similarity between last chunk and the last sentence in the summary, since the cutting point of second to last chunk already determines the boundary of the last chunk.\n3. How are the output generated exactly? The last sentence in 2.4 states each sentence is generated separately. Does it mean each output sentence have different input? How is it different from standard auto-regressive token-by-token generation?\n4. 2.4 should also explicitly refer to Figure. 1 for clarity.\n5. CorDial uses the BART-xsum as initialization, which is trained on XSUM dataset. Are other baselines also gone through the same XSUM training?\n6. What is the model size of all the models in the experiments? It would be better to have some descriptions on model architectures in the experiment section.\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "ACZNsV6ul1p", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes CorDial for abstractive dialogue summarization. CorDial extends BART by generating an intermediate \"summary draft\" which provides weakly-supervised signals and controling the length of the final summary. Results show significant improvements over competitive summarization models such as PEGASUS and BART in multiple different metrics.\n\nSome comments:\n\n1. The paper emphasizes that dialogue summarization is challenging due to its multi-speaker standpoints, casual language, and limited data. Although the use of the proposed summary draft would help solve the first challenge, it is hard to see any correlation between the other problems mentioned and the proposed solutions in the paper. This is especially the case for the controlling of the summary length. Why is this useful specifically for dialogue summarization?\n\n2. The \"summary draft\" is one kind of a content plan, which is widely used in text generation, including text summarization [1]. The technique of extracting key phrase is similar to how content selection is done in [2]. Please compare the proposed solution to other kinds of content planning.\n\n3. To extract key phrases, the method identifies the longest common sub-sequence (LCS) parameterized by a threshold, however how this threshold is set and used is not discussed in the paper. This is important information in order to understand how these key phrases would look like. For example, in Figure 1, how is \"s just one of many boring days at work\" extracted when the LCS is only \"at work\" for turn 2?\n\n[1] https://www.aclweb.org/anthology/C18-1101.pdf\n\n[2] https://arxiv.org/pdf/1808.10792.pdf", "extra_scores": {}, "extra_text": {}}, {"review_id": "6S5mrLlUSyx", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\n<Summary>\n\nThis paper addresses the problem of abstractive dialogue summarization. Its key idea is to label an interrogative pronoun category and extract key phrases from each dialogue turn as weak guide for dialogue summarization. It also proposes a length-controllable generation method for final summary. The proposed approach is evaluated on the SAMSum as one of the largest abstractive dialogue summarization benchmarks, on which it shows competitive performance over recent models. \n\n<Strengths> \n\n1. It proposes a two-step coarse-to-fine approach for abstractive dialogue summarization; it first extracts category labels and key phrases from each dialogue turn, and then generates final summaries by controlling granularity. This idea itself could be novel.\n\n2. It shows strong performance over other recent methods on the recently released SAMSum dataset. \n\n3. It tests the proposed approach with four recent pre-trained language models including DialoGPT, UniLM, PEGASUS and BART.\n\n<Weakness>\n\n1. This paper proposes a novel coarse-to-fine approach for abstractive dialogue summarization but its implementation is largely ad-hoc and engineering intensive and thus bears little technical novelty.\n\n(1) The “coarse” part aims at generating drafts using interrogative pronoun category prediction and key phrase  extraction. \nThese two are largely based on existing techniques (e.g. Ratner et al 2019 and Kitaev & Klien 2018) and some heuristics (e.g. thresholding for key phrases detection). \n\n(2) The “fine” part aims at generating target summary with controllability of granularity. \nIts implementation is also based on a series of engineering heuristics (e.g. dialogue splitting by ROUGE score, binary classification for cutpoint detection). \n\n(3) In summary, it is hard to find methodological novelty in the proposed method.  Given that ICLR is a top premier ML venue, it could be a significant weakness to be a publishable work. \n\n2. Experimental results are rather weak. \n\n(1) Although SAMSum dataset may be one of the best benchmarks for the target task, experiments on only a single dataset is limited to show the generality and effectiveness of the proposed method.\nGiven that the proposed method is ad-hoc, I suspect much additional endeavor may be required to apply to another dataset. \n\n(2) I am not sure whether the comparison in Table 1 is fair enough. Since the proposed approach relies on the additional components for draft construction, it could require more other types of training data or learned modules that other method may not need. This should be clarified in the draft.\n\n<Conclusion>\n\nMy initial decision is ‘reject’ mainly due to lack of technical novelty. Limited experiments could be another issue to be improved. ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a method for the interesting task of dialog summarisation which is slowly getting attention from the research community. In particular, they propose a method which first generates a summary draft and then a final draft.

Pros:
1) The paper is well written
2) Addresses an interesting problem
3) SOTA results

Cons:
1) Lack of novelty
2) No quantitative analysis of the summary draft though it is as an important part of the proposed solution
3) Human evaluations are not adequate (the authors have said they will expand on this but clear details are not provided)
4) The BART model seems to have some advantage as it is pre-trained on XSUM data whereas some of the other models are not (the authors haven't clarified this sufficiently in the rebuttal)

Overall, the reviewers were not completely happy with the work and there was not clear champion. 
Decision: Reject