Abstractive Dialog Summarization with Semantic Scaffolds
Authors: Lin Yuan
Zhou Yu
Venue:  (2020)
Keywords: Abstractive Summarization
Dialog
Multi-task Learning

Abstract: The demand for abstractive dialog summary is growing in real-world applications. For example, customer service center or hospitals would like to summarize customer service interaction and doctor-patient interaction. However, few researchers explored abstractive summarization on dialogs due to the lack of  suitable datasets. We propose an abstractive dialog summarization dataset based on MultiWOZ. If we directly apply previous state-of-the-art document summarization methods on dialogs, there are two significant drawbacks: the informative entities such as restaurant names are difficult to preserve, and the contents from different dialog domains are sometimes mismatched. To address these two drawbacks, we propose Scaffold Pointer Network (SPNet) to utilize the existing annotation on speaker role, semantic slot and dialog domain. SPNet incorporates these semantic scaffolds for dialog summarization. Since ROUGE cannot capture the two drawbacks mentioned, we also propose a new evaluation metric that considers critical informative entities in the text. On MultiWOZ, our proposed SPNet outperforms state-of-the-art abstractive summarization methods on all the automatic and human evaluation metrics.

Under review as a conference paper at ICLR 2020
ABSTRACTIVE DIALOG SUMMARIZATION WITH SEMANTIC SCAFFOLDS
Anonymous authors Paper under double-blind review
ABSTRACT
The demand for abstractive dialog summary is growing in real-world applications. For example, customer service center or hospitals would like to summarize cus- tomer service interaction and doctor- doector-patient interaction. However, few researchers explored abstractive summarization on dialogs due to the lack of suitable datasets. We propose abstractive dialog summarization dataset based MultiWOZ (Budzianowski et al. 2018). we directly apply previous state-of-the-art doc- ument summarization methods on dialogs, there two significant drawbacks: the informative entities such as restaurant names are difficult to preserve, and the contents from different dialog domains sometimes mismatched. To address these two drawbacks, we propose Scaffold Pointer Network (SPNet) to utilize the existing annotation on speaker role, semantic slot and dialog domain. SPNet incorporates these semantic scaffolds for dialog summarization. Since ROUGE cannot capture the two drawbacks mentioned, we also propose new evaluation metric that considers critical informative entities in the text. On MultiWOZ, our proposed SPNet outperforms state-of-the of-the-art abstractive summarization methods all the automatic and human evaluation metrics.
INTRODUCTION
Summarization aims to condense a piece of text to shorter version, retaining the critical informa- tion. On dialogs, summarization has various promising applications in the real world. For instance, the automatic doctor-patient interaction summary can save doctors' massive amount of time used for filling medical records. There is also a general demand for summarizing meetings in order to track project progress in the industry. Generally, multi-party conversations with interactive communica- tion are more difficult to summarize than single-speaker documents. Hence, dialog summarization will be potential field in summarization track.
There are two types of summarization: extractive and abstractive. Extractive summarization selects sentences or phrases directly from the source text and merges them to summary, while abstractive summarization attempts to generate novel expressions to condense information. Previous dialog summarization research mostly study extractive summarization (Murray al., 2005); Maskey & Hirschberg 2005). Extractive methods merge selected important utterances from a dialog to form summary, Because dialogs are highly dependant on their histories, is difficult to produce coherent discourses with set of non-consecutive conversation turns. Therefore, extractive summarization not the best approach to summarize dialogs. However, most modern abstractive methods focus on single-speaker documents rather than dialogs due to the lack of dialog summarization corpora. Popular abstractive summarization dataset like CNN/Daily Mail (Hermann al. (015)) is on news documents. AMI meeting corpus (McCowan et al. 2005) is the common benchmark, but it only has extractive summary.
In this work, we introduce dataset for abstractive dialog summarization based on MultiWOZ (Budzianowski et al. 2018). Seq2Seq models such as Pointer-Generator (See et al., 2017 have achieved high-quality summaries of news document. However, directly applying news summarizer to dialog results two drawbacks: informative entities such as place name are difficult to capture precisely and contents in different domains summarized unequally. To address these problems, we propose Scaffold Pointer Network (SPNet). SPNet incorporates three types of semantic scaffolds in dialog: speaker role, semantic slot, and dialog domain. Firstly, SPNet adapts separate encoder

---

Under review as a conference paper at ICLR 2020
to attentional Seq2Seq framework, producing distinct semantic representations for different speaker roles. Then, our method inputs delexicalized utterances for producing delexicalized summary. and fills in slot values to generate complete summary. Finally, we incorporate dialog domain scaffold by jointly optimizing dialog domain classification task along with the summarization task. We eval- uate SPNet with both automatic and human evaluation metrics MultiWOZ. SPNet outperforms Pointer-Generator (See et al. (017) and Transformer (Vaswani et al. 2017) on all the metrics.
2 RELATED WORK
Rush et al. (013) first applied modern neural models to abstractive summarization. Their approach based on Seq2Seq framework (Sutskever et al., 2014) and attention mechanism (Bahdanau et al.) 2013), achieving state-of-the-art results on Gigaword and DUC-2004 dataset. [uu et al. (2016) 6) pro- posed copy mechanism in summarization, demonstrating its effectiveness by combining the advan- tages of extractive and abstractive approach. [See et al. (2117) applied pointing (Vinyals et al., 2015) as copy mechanism and use coverage mechanism (Tu et al. 2016) to discourage repetition. Most recently, reinforcement learning (RL.) has been employed in abstractive summarization, RL-based approaches directly optimize the objectives of summarization (Ranzato et al. 2016 Celikyilmaz et al. 2018). However, deep reinforcement learning approaches are difficult to train and more prone to exposure bias (Bahdanau et al. 2017).
Recently, pre-training methods are popular in NLP applications. BERT (Devlin et al. 2018) and GPT (Radford al., al.] 2018)) have achieved saute-of-the-eeee performance many tasks, including sum- marization. For instance, Zhang et al. (2019) proposed method to pre-train hierarchical document encoder for extractive summarization. Hooag et al. (2019) proposed two strategies to incorporate a pre-trained model (GPT) to perform the abstractive summarizer and achieved better performance. However, there has not been much research on adapting pre-trained models to dialog summarization.
Dialog summarization, specifically meeting summarization, has been studied extensively. Previous work generally focused on statistical machine learning methods in extractive dialog summarization: Galley (006) used skip-chain conditional random fields (CRFs) (afferrty et al., (001)) rank-
ing method in extractive meeting summarization. Waan & Cardic (2013) compared support vector
machines (SVMs) (Cortes & Vapnik [9955 with LDA-based topic models (Blei et al., 2003) for
producing decision summaries. However, abstractive dialog summarization was less explored due to the lack of suitable benchmark. Recent work (Wang & Cardie 2016 Goo & Chen 2018 Pan
iet al. 2018) created abstractive dialog summary benchmarks with existing dialog corpus. Goo & Chen (2018) annotated topic descriptions in AMI meeting corpus as the summary. However, topics they defined are coarse, such as "industrial designer presentation". They also proposed model with sentence- sentence-gated mechanism incorporating dialog acts to perform abstractive summarization. More- over, [i et al. (019)) first built model to summarize audio-visual meeting data with an abstractive method. However, previous work has not investigated the utilization of semantic patterns dialog, so we explore it in-depth in our work.
PROPOSED METHOD
As discussed above, state-of-the-art document summarizers are not applicable in conversation set- tings. We propose Scaffold Pointer Network (SPNet) based on Pointer-Generator (See al. 2017). SPNet incorporates three types of semantic scaffolds to improve abstractive dialog summarization: speaker role, semantic slot and dialog domain.
3.1 BACKGROUND
We first introduce Pointer-Generator (See et al. (017). is a hybrid model of the typical Seq2Seq at- tention model (Nallapati et al.) 2016) and pointer network (Vinyals et al. 2015). Seq2Seq framework encodes source sequence and generates the target sequence with the decoder. The input sequence is fed into the encoder token by token, producing the encoder hidden states h, in each encoding step. The decoder receives word embedding of the previous word and generates a distribution to decide attention
the target element in this step, retaining decoder hidden states 8t. In
2

---

Under review as a conference paper at ICLR 2020
distribution a' is computed in Bahdanau et al. (2015):
e v² tanh (Wih, + W,st + batin)
a' softmax (e')
where Wh. W,, and buttn are all learnable parameters.
With the attention distribution a', context vector hy computed as the weighted sum of encoder's hidden states. Context vector is regarded as the attentional information in the source text:
n Σ a
GenerrrG differs from typical Seq2Seq attention model in the generation process. The point- ing mechanism combines copying words directly from the source text with generating words from fixed vocabulary. Generation probability Pgen is calculated as "a soft switch" to choose from copy and generation:
Pgen (wf. w/ st w bper)
(3)
where Ie is the decoder input, Wh,, Was wy and bptr are all learnable parameters. is sigmoid function, so the generation probability Pgen has a range of [0, 1].
The ability to select from copy and generation corresponds to a dynamic vocabulary. Pointer net- work forms an extended vocabulary for the copied tokens, including all the voc.-ouaaallayOOO00 words appeared in the source text. The final probability distribution P(w) extended vocabulary is computed as follows:
Pvocab softmax (V' (V [st, /st, h;] 8')
P(w)= Pgen P Pvocab (w) (1 Pgeu) Σ af
(4)
where Procab the distribution on the original vocabulary, V", V, b and B are learnable parameters used to calculate such distribution.
3.2 SCAFFOLD POINTER NETWORK (SPNET)
Our Scaffold Pointer Network (depicted in Figure D based on Pointer-Generator (See et al. 2017). The contribution of SPNet is three-fold: separate encoding for different roles, incorporating semantic slot scaffold and dialog domain scaffold.
3.2. SPEAKER ROLE SCAFFOLD
Our encoder-decoder framework employs separate encoding for different speakers in the dialog- User utterances or!er and system utterances =zye are fed into user encoder and system encoder separately to obtain encoder hidden states hear and 4, The attention distributions and context vectors are calculated as described in section 3. In order to merge these two encoders in our framework, the decoder's ' hidden state sp is initialized as:
8p concat(hye" .h1
The pointing mechanism our model follows the Equation
3 and we obtain the context vector h;:
h concat
ag
h
a
syss
h sys
3.2.2 SEMANTIC SLOT SCAFFOLD
We integrate semantic slot scaffold by performing delexicalization original dialogs. Delexicaliza- tion is a common pre-pree processing step in dialog modeling. Specifically, delexicalization replaces the slot values with its semantic slot name(e.g, replace 18:00 with [time]). It is easier for the language modeling to process delexicalized texts, as they have a reduced vocabulary size. But these generated sentences lack the semantic information due the delexicalization. Some previous dialog system
3

---

Under review as a conference paper at ICLR 2020
Delexicalized Slot Filling
18:00
18:00 Final
16:00
Distribution
[tme]]
200
Lexical Value Matcher
Vocabulary Distribution
Abstractive Summarization
x (1
Pgen)
ggen
x P
vocab
Attention Distribution
[Liee]
Pgen
User Encoder
Hidden States
Decoder Hidden States
Context Vector
[time]
[time]
18:00
System Encoder
Hidden States
Domain Classification
have
it
later
Figure 1: SPNet overview. The blue and yellow box is the user and system encoder respectively. The encoders take the delexicalized conversation as input. The slots values are aligned with their slots position. Pointing mechanism merges attention distribution and vocabulary distribution to obtain final distribution. then fill slots values into slot tokens to convert template to complete summary. SPNet also performs domain classification to improve encoder representation.
research ignored this issue (Wen et al. 2015) completed single delexicalized utterance (Sharma et al., 2017) as generated response. We propose to perform delexicalization in dialog summary, since delexicalized utterances slots with the copy and pointing mechanism.
simplify dialog modeling. We fill the generated templates with
We first train the model with the delexicalized utterance. Attention distribution a' the source tokens instructs the decoder to fill up the slots with lexicalized values:
value(w (W stot)
value(w:) elot dol(w.)==atot
Note that Wator specifies the tokens that represents the slot name (e.g. [hotel place], [time]). Decoder directly copies lexicalized value value(w:)) conditioned on attention distribution of. If w is not slot token, then the probability P(w) is calculated as Equation
3.2.3 DIALOG DOMAIN SCAFFOLD
We integrate dialog domain scaffold through multi-task framework. Dialog domain indicates different conversation task content, for example, booking hotel, restaurant and taxi in MultiWOZ dataset. Generally, the content in different domains varies so multi-domain task summarization more difficult than single-domain. We include domain classification the auxiliary task incorpo- rate the prior that different domains have different content. Feedback from the domain classification task provides domain specific information for the encoder to learn better representations. For do- main classification, we feed the concatenated encoder hidden state through binary classifier with two linear layers, producing domain probability d. The tth element d, in represents the probability
4

---

Under review as a conference paper at ICLR 2020
of the 7th domain:
d= (R h ba)) b())
(8)
where U. U", ba and b/ are all trainable parameters in the classifier. We denote the loss function of summarization loss] and domain classification loss2. Assume target word timestep is wi. lo88; is the arithmetic mean of the negative log likelihood of wi over the generated sequence:
T
1
loss
log P(w")
T
The domain classification task is a multi-label binary classification problem. We use binary cross entropy loss between the tt domain label d, and predict probability d, for this task:
D
loss
de log d, (1 d.) log (1 -d,)
(10)
D
where |D] is the number of domains. Finally, reweight the classification loss with hyperparame- ter and the objective function is:
loss lossi Aloss2
(11)
4 EXPERIMENTAL SETTINGS
4.1 DATASET
We validate SPNet on MultWWO--2.0 dataset (Budzianowski et al. 2018). MultiWOZ consists of multi-domain conversations between tourist and information center clerk varies booking tasks or domains, such as booking restaurants, hotels, taxis, etc. There are 10,438 dialogs, spanning domains. 3,406 of them single-domain (8.93 turns average) and 7.302 multi- domain (15.39 turns on average). During MultiWOZ data collection, instruction is provided for crowd workers perform the task. We use the instructions the dialog summary, and example data is shown in Table 2 Dialog domain label is extracted from existing MultiWOZ, annotation. In the experiment, we split the dataset into 8,438 training, 1,000 validation, and 1,000 testing.
4.2 EVALUATION METRICS
ROUGE Lin 2004) is standard metric for summarization, designed to measure the surface word alignment between a generated summary and a human written summary. We evaluate our model with ROUGE-1, ROUGE-2 and ROUGE-L. They measure the word-overlap. bigram-overlap, and longest common sequence between the reference summary and the generated summary respectively. We obtain ROUGE scores using the files2rouge package]] However, ROUGE is insufficient to measure summarization performance. The following example shows its limitations:
Reference: You are going [restaurant name] at [time]. Summary: You are going to [restaurant_name] at.
In this case, the summary has high ROUGE score, as it has a considerable proportion of word overlap with the reference summary. However, it still has poor relevance and readability, for leaving out one of the most critical information: [time]. ROUGE treats each word equally in computing n n-gram overlap while the informativeness actually varies: common words or phrases (e.g. "You are going to"), significantly contribute the ROUGE score and readability, but they almost irrelevant to essential contents. The semantic slot values (e.g. [restaurant name], [time]) are more essential compared other words in the summary. However, ROUGE did not take this into consideration. To address this drawback in ROUGE, we propose a new evaluation metric: Critical Information Completeness (CIC). Formally, CIC is a recall of semantic slot information between candidate summary and reference summary. CIC is defined as follows:
Count (v) VEV
Σ
CIC
(12)
5

---

Under review as a conference paper at ICLR 2020
Models
ROUGE-1
ROUGE-2
ROUGE-L
CIC
base (Pointer-Gen) (See al. 2017
62.89
48.61
59.30
42.47
Transformer (Vaswanı et al. 2017 base + speaker role
63.12
50.63
61.04
42.84
72.01
60.55
68.40
53.08
base + speaker role + semantic slot
90.68
83,54
84.36
70.25
SPNet (base + speaker role semantic slot dialog domain)
90.97
84.14
85.00
70.45
Automatic evaluation results on MultiWOZ. We use Pointer-Generator as the base model and gradually add different semantic scaffolds.
Table I:
where stands for set of delexicalized values in the reference summary, Count (v) is the number of values co-occurring in the candidate summary and reference summary, and is the number values in set V. In our experiments, CIC is computed as the arithmetic mean over all the dialog domains to retain the overall performance.
CIC a suitable complementary metric to ROUGE because accounts for the most important infor- mation within each dialog domain. CIC can be applied to any summarization task with predefined essential entities. For example, in news summarization the proper nouns are the critical information to retain.
4.3 IMPLEMENTATION DETAILS
We implemented our baselines with OpenNMT framework (Klein et al. 2017). We delexicalize utterances according the belief span annotation. To maintain the generalizability of SPNet, we combine the slots that refer to the same information from different dialog domains into one slot (e.g. time). Instead of using pre-trane pre-trained word embeddings like GloVe (Pennington et al. 2014). we train word embeddings from scratch with 128-dimension embedding layer. We set the hidden states of the bidirectional LSTM encoders to 256 dimensions, and the unidirectional LSTT decoder to 512 dimension. Our model is optimized using Adam (Kingma & Ba (014)) with learning rate of 0.001, 31 = 0.9, 82 = 0.999. We reduce the learning rate to half to avoid overfitting when the validation loss increases. We set the hyperparameter À to 0.5 the objective function and the batch size to eight. We use beam search with beam size of three during decoding. We use the validation set to select the model parameter. Our model with and without multi-task takes about 15 epochs and seven epochs to converge, respectively.
5 RESULTS AND DISCUSSIONS
5.1 AUTOMATIC EVALUATION RESULTS
To demonstrate SPNet's effectiveness, we compare it with two state-of-the-art methods, Pointer- Generator (See et al., 2017) and Transformer (Vaswani al. 2017). Pointer-Generator is the state- inference, we use length penalty and
of-the-art method abstractive document summarization. coverage penalty mentioned in Gehrmann al. 2018). The hyperparameters in the original imple- mentation (See et al. 2017) were used. Transformer uses attention mechanisms to replace recurrence for sequence transduction. Transformer generalizes well many sequence-to-sequuence problems, so we adapt it to our task, following the implementation in the official OpenNMT-py documentation.
We show all the models' results in Table We observe that SPNet reaches the highest score in both ROUGE and CIC. Both Seiner--Gnerator and Transformer achieve high ROUGE scores, but rel- ative low CIC scores. It suggests that the baselines have more room for improvement on preserving critical slot information. All the scaffolds we propose can be applied to different neural network models. In this work we select Döinter-Generato as our base model in SPNet because we observe that Transformer only has small improvement over Cennter-Generatrr but is having a higher cost on training time and computing resources. We observe that SPNet outperforms other methods all the automatic evaluation metrics with big margin, as it incorporates all the three semantic scaffolds. Semantic slot contributes the most to SPNet's 's increased performance, bringing the largest increase on all automatic evaluation metrics.

---

Under review as a conference paper at ICLR 2020
Conversation
Client: I'm looking for a place to stay. It needs to be guesthouse and include free with.
Help desk: There are 23 hotels that meet your needs. Would you like to narrow your search by area and/r price range"
Client: would like
cheap include free parking.
Help desk: There are nine guesthouse hotels various areas. What part town you hoping, for?
Client: Nothing in particular. just need it booked for people for a total of 4 nights starting from sunday. I'd also like the reference number. pleas.
Help desk: Sure! We will book you hotel with stars. The reservation will 6 people for nights How many rooms would you like to reserve?
Client: Well, six people suppose be needing three noems, four nights, starting Sunday,
Help desk: You'le 'e booked at the Alexander Bed and Breakfast, 517a coldham lane, for 6 people for four nights starting Sunday. Your reforence number is JXVKZ7KV.
Client: Thank you! I'm looking for restaurant. Ideally an Italian place same price range in the centre.
Help desk: I found Ask restaurant would you like book that?
Client: Is the name of the restaurant Ask restaurant?
Help desk: Yes, restaurant called Ask restaurant. Would you like book table there?
Client: Yes. Book for the same group of people un the same day at 18:45. Can get the reference number please?
Help desk: Great! Your booking for people at the restaurant Ask on Sunday at 18:45 was successful! Your reference number
Y2YSQYBY.
Client: am all sot. Thank you.
Help desk:
Thank you for using the Cambridge Townelnfo Help Desk!
Summaries
Pointer-Generator:
are traveling Cambrrdde and looking forward ity local restaurants. You are looking for place to stay. hotel should include free wifi and should be in the type of guesthouse. The hotel does n't need to include internet. If there is no such hotel, how ahout one that has free wifi. Once you find
hotel you want book it 6 people and nights starting from Sunday. If the booking fails how about nights. Make sure you get the reference number. You also looking for information particular restaurant. Its name called Ask restaurant restaurant. Make you you the price range and postcode.
SP:Net:
are looking information Cambridge, are looking a place to stay. hotel should be type of guest house should include free wifi, The hotel should he in the cheap price range. Once you find the hotel you want to book it for 6 people and 4 nights starting from Sunday, Make sure you get the referenee number. You are also looking for restaurant. The restaurant should be the sume price range the hotel and should serve Italian Food. The restaurant should in the same area the hotel. Once you find the restaurant you want to book a table for the same group of people at 18:45 on the same day. Make sure you get the reference number.
Ground truth:
place to stay. hotel should include free wifi and should in the type of guest house. The hotel should be in the cheap price range and should include free parking. Once you find the hotel you want book for people and 4 nights starting from Sunday, Make sure you get the reference number. You are also looking for restaurant. The restaurant should be the you find the restaurant you want to book table for the same group of people at 18:45 on the same day. Make sure you got the reference number.
are planning your trip Cambridge, You are looking
price range the botel and should be the centre. The restaurant should serve italian food. Once
Table 2: An example dialog and Pointer-Generate SPNet and ground truth summaries. We un- derline semantic slots in the conversation. Red denotes incorrect slot values and green denotes the correct ones.
5.2 HUMAN EVALUATION RESULTS
We also perform human evaluation to verify if our method's increased performance on automatic evaluation metrics entails better human perceived quality. We randomly select 100 test samples from MultiWOZ test set for evaluation. We recruit 150 crowd workers from Amazon Mechanical Turk. For each sample, we show the conversation, reference summary, as well as summaries generated by Center-Generator and SPNet to three different participants. The participants are asked to score each summary on three indicators: relevance, conciseness and readability on summary pair (tie allowed).
scale, and rank the
We present human evaluation results in Table 3 In the scoring part, our model outperforms Pointer- Generator in all three evaluation metrics. SPNet scored better than Pointer-Generator on relevance and readability. All generated summaries are relatively concise; therefore, they score very similar in conciseness. Ground truth is still perceived as more relevant and readable than SPNet results. However, ground truth does not get a high absolute score. From the feedback of the evaluators, we found that they think that the ground truth has not covered all the necessary information in the con- versation, and the description is not so natural. This motivates us to collect a dialog summarization dataset with high- human- show more differences between different summaries. SPNet outperforms Generatr with a large margin. Its performance is relatively close to the ground truth summary.
summaries in the future. Results in the ranking evaluation
7

---

Under review as a conference paper at ICLR 2020
Summary
Relevance
Conciseness 3.67
Readability
Ground truth
3.83
3.87
Pointer-Gen See et al. 2017)
3.56
3.58
3.64
SPNet
3.77
3.67
3.80
Rank Pair
Win
Lose
Tie
SPNet vs. Pointer-Geen
61.3
30.3
8.3
SPNet vs. Ground truth
32.3
48.0
19.7
Table 3: The upper is the scoring part and the lower is the the ranking part. SPNet outperforms Pointer-Generator in all three human evaluation metrics and the differences are significant, with the confidence over 99.5% in student test. In the ranking part, the percentage of each choice is shown in decimal. Win, lose and tie refer the state of the former summary ranking.
5.3 CASE STUDY
Table[] an example summary from all models along with ground truth summary. We observe that Pointer-Ceenerator ignores some essential fragments, such as the restaurant booking information (6 people, Sunday, 18:45). Missing information always belongs to the last several domains (restau- rant in this case) in a multi-domain dialog. We also observe that separately encoding two speakers reduces repetition and inconsistency. For instance, Pointer-Generator' summary mentions "free wifi" several times and has conflicting requirements wifi. This is because dialogs has informa- tion redundancy, but single-speaker model ignores such dialog property.
Our method has limitations. In the example shown in Table 2] our summary does not mention the hotel name (Alexander Bed and Breakfast) and its address (517a Coldham Lane) referred in the source. It occurs because the ground truth summary doe not cover it in the training data. As supervised method, SPNet hard to generate a summary containing additional information beyond the ground truth. However, in some cases, SPNet can also correctly summarize the content not covered in the reference summary (see Table in Appendix).
Furthermore, although our SPNet achieves much-improved performance, the application of SPNet still needs extra annotations for semantic scaffolds. For dialog dataset, speaker role scaffold is natural pattern for modeling. Most multi-domain dialog corpus has the domain annotation. While for texts, for example news, its topic categorization such as sports or entertainment can be used as domain annotation. We find that semantic slot scaffold brings the most significant improvement, but it is seldom explicitly annotated. However, the semantic slot scaffold can be relaxed to any critical entities in the corpus, such as team name in sports news or professional terminology in a technical meeting.
CONCLUSION AND FUTURE WORK
We adapt dialog generation dataset, MultiWOZ to abstractive dialog summarization dataset. We propose SPNet, an end-to-end model that incorporates the speaker role, semantic slot and dialog domain the semantic scaffolds to improve abstractive summary quality. We also propose an automatic evaluation metric CIC that considers semantic slot relevance to serve as a complementary metric to ROUGE. SPNet outperforms baseline methods in both automatic and human evaluation metrics. It suggests that involving semantic scaffolds efficiently improves abstractive summarization quality in the dialog scene.
Moreover, we can easily extend SPNet to other summarization tasks. We plan to apply semantic slot scaffold to names or location names to ensure that they are captured correctly in the generated summary. We also plan to collect numan-human dialog dataset with more diverse human numan-writen summaries.
summarization. Specifically,
annotate the critical entities such person
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning align and translate. In ICLR 2015: International Conference Learning Represen- tations 2015, 2015.
Dzmitry Bahdanau, Philemon Brakel, Kelvin Xu, Anirudh Goyal, Ryan Lowe, Joelle Pineau, Aaron C. Courville, and Yoshua Bengio. An actor-critic algorithm for sequence prediction. In ICLR 2017: International Conference on Learning Representations 2017, 2017.
David M. Blei, Andrew Y. Ng. and Michael I. Jordan. Latent dirichlet allocation. Journal of Machine Learning Research, 3:993-1022, 2003.
Pawe Budzianowski. Tsung-Hsien Wen, Bo-Hsiang Tseng, ligo Casanueva, Stefan Ultes, Osman Ramadan, and Milica Gai. Multiwoz - a large-scale multi-domain wizard-of-oz dataset for task- 2018.
oriented dialogue modelling. arXiv preprint
Asli Celikyilmaz, Antoine Bossclut, Xiaodong He, and Yejin Choi. Deep communicating agents for abstractive summarization. In NAACL HLT 2018: 16th Annual Conference of the North Ameri- can Chapter of the Association for Computational Linguisties: Human Language Technologies, volume 1, pp. 1662-1675, 2018.
Corinna Cortes and Vladimir Vapnik. Support-vector networks. Machine Learning, 20(3):273-297, 1995.
Jacob Devlin, Ming-Wet Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint 1rXiv:1810.048// 2018.
Michel Galley, skip-chain conditional random field for ranking meeting utterances by importance. In Proceedings of the 2006 Conference on Empirical Methods in Natural Language Processing, pp. 364-372, 2006.
Sebastian Gehrmann. Yuntian Deng, and Alexander M. Rush. Bottom-up abstractive summarization. In EMNLP 2018: 2018 Conference on Empirical Methods in Natural Language Processing, pp- 4098 4109, 2018
Chih- Wen Goo and Yun-Nung Chen. Abstractive dialogue summarization with sentence- gated mod- eling optimized by dialogue acts. arXiv preprint arXiv: 1809.00500 2018.
Jiatao Gu, Zhengdong Lu, Hang Li, and Victor O.K. Li. Incorporating copying mechanism in seguence-to-sequeee learning. Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pp. 1631-1640, 2016.
Karl Moritz Hermann, Tom Koisk, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suley- man, and Phil Blunsom. Teaching machines to read and comprehend. In NIPS' Proceedings of the 28th International Conference on Neural Information Processing Systems Volume 1, pp- 1693-1701, 2015.
Andrew Hoang, Antoine Bosselut, Asli Celikyilmaz, and Yejin Choi. Efficient adaptation of pre- trained transformers for abstractive summarization, arXiv preprint arXiv:1906.00138, 2019.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 142.6664, 2014.
Guillaume Klein, Yoon Kim, Yuntian Deng, Jean Senellart, and Alexander M. Rush. Opennmt: Open-source toolkit for neural machine translation. In Proceedings of ACL 2017, System Demon- strations, pp. 67-72, 2017.
John D. Lafferty, Andrew McCallum, and Fernando C. N. Pereira. Conditional random fields: Probabilistic models for segmenting and labeling sequence data, In ICML. '01 Proceedings of the Highteenth International Conference Machine Learning, pp. 282-289, 2001.
Manling Li, Lingyu Zhang, Heng Ji, and Richard J. Radke. Keep meeting summaries topic: Abstractive multi-modal meeting summarization. ACL 2019 : The 57th Annual Meeting of the A Association for Computational Linguistics, pp. 2190-2196, 2019.

---

Under review as a conference paper at ICLR 2020
Chin- Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text Summarization Branches Out: Proceedings of the ACL-04 Workshop, pp. 74-81, 2004.
Sameer Maskey and Julia Hirschberg. Comparing lexical, acoustic/prosodie,, structural and dis-
course features for speech summarization. INTERSPEECH, pp. 221-624, 2005.
I. McCowan, J. Carletta, W. Kraaij, S. Ashby, S. Bourban, M. Flynn, M. Guillemot, T. Hain, Kadlec, V. Karaiskos, M. Kronenthal, G. Lathoud, M. Lincoln, A. Lisowska, W. Post, Den- nis Reidsma, P. Wellner, L.P.J. Noldus, F Grieco, L.W.S. Loijens, and P.H. Zimmerman. The ami meeting corpus. Symposium on Annotating and Measuring Meeting Behavior, pp. 137-140, 2005.
Gabriel Murray, Steve Renals, and Jean Carletta. Extractive summarization of meeting recordings. In INTERSPEECH, pp. 593-596, 2005.
Ramesh Nallapati, Bowen Zhou, Ccero Nogueira dos Santos, aglar Glehre, and Bing Xiang. Ab- stractive text summarization using 20th SIGNLL Conference on Computational Natural Language Learning, pp. 280-290, 2016.
mns and beyond. In Proceedings The
Haojie Pan, Junpei Zhou, Zhou Zhao, Yan Liu, Deng Cai, and Min Yang. Dial2desc: End-to-end dialogue description generation. arXiv preprint arXiv: 10t1100110005 2018.
Jeffrey Pennington, Richard Socher, and Christopher D. Manning. Glove: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Lan- guage Processing (EMNLP), pp. 1532-1543, 2014.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language un- derstanding by generative pre-training. URL hutps://s3-s--wettt amazonaws. com/openai- understanding paper. pdf. 2018.
Marc' Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojeiech Zaremba. Sequence level train- ing with recurrent neural networks. In ICLR 2016 International Conference on Learning Rep- resentations 2016, 2016.
Alexander M. Rush, Sumit Chopra, and Jason Weston. A neural attention model for abstractive sentence summarization. arXiv preprint 1rXiv:/509.000 50680000005,,.. 2015.
Abigail See, Peter J. Liu, and Christopher D. Manning. Get to the point: Summarization with pointer-generator networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), volume 1, pp. 1073-1083, 2017.
Shikhar Sharma, Jing He, Kaheer Suleman, Hannes Schulz, and Philip Bachman. Natural language generation in dialogue using lexicalized and delexicalized data. In International Conference Learning Representations (ICLR) Workshop, 2017.
Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. Sequence to sequence learning with neural networks. In Advances Neural Information Processing Systems 27, pp. 3104-3112, 2014.
Zhaopeng Tu, Zhengdong Lu, Yang Liu, Xiaohua Liu, and Hang Li. Modeling coverage for neural machine translation. arXiv preprint (001:1600..48811 2016.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances Neural Information Processing Systems, pp. 5998-6008, 2017.
Oriol Vinyals, Meire Fortunato, and Navdeep Jaitly. Pointer networks. In NIPS'I5 15 Proceedings of the 28th International Conference Neural Information Processing Systems Volume 2, pp- 2692-2700, 2015.
Lu Wang and Claire Cardie. Domain-independee independent abstract generation for focused meeting sum- marization. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1395-1405, 2013.
10

---

Under review as a conference paper at ICLR 2020
Lu Wang and Claire Cardie. arXiv:1606.07965. 2016.
Summarizing decisions in spoken meetings.
arXiv preprint
Tsung-Hsien Wen, Milica Gasic, Dongho Kim, Nikola Mrksic, Pei hao Su, David Vandyke, and Steve Young. Stochastic language generation in dialogue using recurrent neural networks with convolutional sentence reranking. In Proceedings of the 16th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pp. 275-284, 2015.
Xingxing Zhang, Furu Wei, and Ming Zhou. Hibert: Document level pre-training of hierarchical bidirectional transformers for document summarization. In ACL 2019 The 57th Annual Meeting of the Association for Computational Linguistics, pp. 5059-5069, 2019.

---

Under review as a conference paper at ICLR 2020
A SUPPLEMENT TO CASE STUDY
Supplement Summary
Transformer: You are planning your trip in Cambridge. You looking for place stay. The hotel doesn't 't need to include internet and should include free parking. The hotel should in type of guest- house. If there no such hotel, how about one that is in the moderate price range? Once you find the hotel, you want to book it for 6 people and nights starting from Sunday. Make sure you get the reference number. You also looking forward dine. The restaurant should be in the centre. Make you get the reference number.
Human Evaluation
Which one is better in your opinion, summary Evaluator: Summary (SPNet)
summary 2 (SPNct) tie?
What is the reason for your rank? We recommend you to talk about it from relevance, conciseness and readability.
Evaluator: Summary is more coherent and relevant. is just concise but readable.
Which one is better in your opinion, summary (SPNet) or summary (Ground Truth) or tie?
Evaluator: Tie
What is the reason for your rank? We recommend you talk about it from relevance, conciseness and readability.
Evaluator: Summary and are almost identical. They are equally coherent, relevant, readable.
Table 4: Supplement to the case in Table 2 The summary generated by Transformer is shown in supplement summary. Red denotes incorrect slot values and green denotes the correct ones. Human Evaluation part provides the evaluator's 's choice and feedback in ranking summary pairs. Content in the brackets is not shown to the evaluators.
12

---

Under review as a conference paper at ICLR 2020
B DIALOG SUMMARIZATION CASES
Conversation
Client: I'm looking for a moderately priced place to eat that's the centre town.
Help desk: What type of cuisine are you looking for? There are 21 restaurants in that area.
Client: don't have preference.
Help desk: Out the 21 restaurant choices, one is the Yippee Noodle Bar which is moderately priced in the centre of town. Would you like to make a reservation?
Client: That sounds great, what is the postcode?
Help desk: Their postcode is eb1 Ilh and if you want reservation just let me know the time, party size, and day and I'll make for you.
Client: Not now, thanks, Here's weird one, though are there any expensive O-star hotels in center of town?
Help desk: I'm sorry, but there are expensive O-star hotels in the area. Can help you find something
else?
Client:
Do you have a hotel in the moderate price range?
Help desk:
The Cityroomz Hotel is moderatly priced, and has free internet and parking. Would you like to make a reservation now?
Client: That sounds nice, yes please book room for 3 guests staying for 5 nights starting on Sunday,
Help desk: I'm sorry, those particular dates are filled, if you'd like could try make the trip shorter or maybe change days?
Client: How about just for the one night?
Help desk: Alright, I able to book your hotel room at the Cityroomz, for people, for one night. Your reference number is E62IXZGB.
there anything clse can do?
Client: will also need a taxi to go from the restaurant to the hotel, I want to leave the restaurant at 02:15.
Help desk: I have booked your taxi. It will be blue volvo and the contact number is 07729446320.
Client: Great. Thanks. That's all I need today.
Help desk: You're welcome. Have wonderful visit.
Summaries
Pointer-Generator:
You are traveling to Cambridge. You are looking for a restaurant. The restaurant should serve food and should be in the moderate price range. Make sure you get postcode and address. You are also looking for information in the centre. The hotel should have star of 0 and should include free wifi. The hotel doesn't t need to include internet. If there is no such hotel, how about that doesn't 't have free parking? Once you find hotel, you want to book it for people and nights starting from Sunday, If the booking fails, how about night? Make sure you get the reference number.
SPNet:
You are traveling to Cambridge and looking forward try local restaurants. You are looking for a place to dine. The restaurant should be in the moderate price range and should be in the contre. Make sure you get postcode. You are also looking for a place to stay. The hotel should be in the type of hotel and should have O-star. The hotel should be in the same area as the restaurant and should be in the same price range as the restaurant. Once you find the hotel, you booking fails, how about night? Make sure you get the reference number. You also want to book taxi to commute between the 2 places. You want to leave the restaurant by 02:15. Make sure you get contact number and car type.
book it for 3 people and nights starting from Sunday. If
Ground Truth:
You are traveling to Cambridge and looking forward to try local restaurants. The restaurant should be in the centre and should be in the moderate price range. Make sure you get postcode. You are also looking for place to stay. The hotel should be in the expensive price range and should have star of O. The hotel should be in the same area as the restaurant. If there is such hotel, how about one that in the moderate price range? Once you find the hotel you want to book the booking fails how about night. Make sure you get the reference number. You also want to book taxi to commute between the 2 places. You want to leave the restaurant by 02:15. Make sure you get contact number and car type.
people and 5 nights starting from Sunday. If
Table An example dialog and Pointer-Generator SPNet and ground truth summaries. The dialog spans over three domains: restaurant, hotel and taxi. We underline semantic slots in the conversation. Red denotes incorrect slot values and green denotes the correct ones.
13

---

Under review as a conference paper at ICLR 2020
Conversation
Client:
would like a British food restaurant in the centre.
Help desk: Sure, there are 7 restaurants that meet your needs. 4 are moderate and 3 are expensive. Do you have preference?
Client: Only the best for my family We'll take the expensive onc. Book us table for 5 14:00 Thursday.
Help desk: I'm sorry I am having difficulty making reservation for you. Shall we try another time or restaurant type?
Client: Let's try Italian instead.
Help desk: Caffe Uno is a very nice. expensive Italian restaurant in the center of town. Would you like table there?
Client: Actually, change my mind. think want to stick with British food after all. Can you suggest any one that's the centre of town?
Help desk: Fitzbillies Restaurant, the Cambridge Chop House, and Midsummer House are all centrally located, British, and very expensive, but none have a table for 5 available on Thursday at 14:00.
Client: Can you try the Cambridge Chop House for Thursday for 5 people 13:00 instead?
Help desk: Your reservation at the Cambridge Chop House for people at 13:00 on Thursday was success- ful. Your reference number is U6GVSZZV. Is there anything else I can help with today?
Client: No, that's all I need. Thanks for your help!
Help desk: You're welcome, thank you for using our system.
Client:
You're re welcome! was great!
Help desk: Have a good day!
Summaries
Pointer-Generator:
You are looking for restaurant. The restaurant should serve British food and should be in the centre. Once you find the restaurant you want to book table for 5 people at 14:00 Thursday. Make sure you get the reference number.
SPNet:
You are looking for a place to dine. The restaurant should serve British food and should be in the centre. The restaurant should be in the expensive price range. there such restaurant, how about one that serves British food? Once you find the restaurant you want book table people 13:00 on Thursday, the booking fails, how about different restaurant in the same area and price range? Make sure you get the reference number.
Ground Truth:
You are looking for restaurant. The restaurant should serve British food and should be in the centre. Once you find the restaurant, you want to book a table for 5 people at 14:00 on Thursday. If the booking fails, how about 13:00? Make sure you get the reference number.
Table An example dialog and Gennterr SPNet and ground truth summaries. The dialog spans over one domain: restaurant. We underline semantic slots in the conversation. Red denotes incorrect slot values and green denotes the correct ones. Blue denotes the content not covered by ground truth in SPNet's summary.
14


Reviews:
[{"review_id": "S1e6F8NOtB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper \"Abstractive Dialog Summarization with Semantic Scaffolds\" presents a new architecture that the authors claim is more suited for summarizing dialogues. The dataset for summarization was synthesized from an existing conversation dataset. \n\nThe new architecture is a minor variation of an existing pointer generator network presented by See et al. First the authors used two different sets of parameters to encode the user and the system responses. The authors also pre-process the dialog by replacing the slot values by their slot keys. Finally, the authors use an auxiliary task of detecting the domain of the dialog. \n\nThese three different enhancements are all called \"scaffolds\" by the authors, hance the title of the paper.\n\nThis paper is not suited for ICLR because of its limited novelty. The three enhancements proposed by the authors are long known and incremental. ", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "HJWCJKttH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Authors proposed an enhanced Pointer-Generator model called SPNet. The key difference between SPNet and PG are the separate handling or using of speaker role, semantic slot and domain labels. Authors also proposed a new metrics called Critical Information Completeness (CIC) to address ROUGE's weakness in assessing if key information is missing in the output.\n\nSPNet considers speak role by using separate encoders for each speaker in the dialog. The hidden state vectors of all speakers are concatenated for next layer. \n\nSemantic slot is modeled by delexicalizing the input, i.e. replacing values (18:00) with their semantic category (time). The actual value is later recovered from input text by copying over the corresponding raw tokens according to the attention layer. The domain labels are incorporated by combining categorization task loss into the final training loss.\n\nAuthors used the MultiWoz dataset to evaluate the model and compared it with state-of-the-art Pointer-Generator and Transformer models. ROUGE and proposed CIC metrics all show clear improvements in SPNet. The best performance was observed when all three improvements over SPNet are leveraged. Authors also provided example generated summary and discussed the difference between SPNet PG and baseline. An additional human evaluation was conducted which confirmed the quality gain.\n\nThe main concern of Reviewer is the inconsistency in the paper.\n\n1) Authors claimed to \"propose an abstractive dialog summarization dataset based on MultiWOZ (Budzianowski et al., 2018)\" in the abstract and introduction, which sounds like part of their contribution is creating a new dataset, but in experiment section there's no discussion about how the dataset was created or used at all. The same claim reappeared as the first sentence in the conclusion section.\n\n2) Authors emphasized two drawbacks in the beginning of the paper, but didn't discuss or show any evidence of those drawbacks from data later.\n\nThe above inconsistency suggests the paper may not be quite ready for publication.\n\nOther issues found by Reviewer:\n\n1) In equation (7), value() seems to be the word while on the right hand side it's a numerical value (max a_i^t). Did Authors mean argmax?\n\n2) In Table 1, dialog domain seems to provide very marginal improvement, does it justify the complexity added?\n\n3) In Section 4.3, why do we need to train a customized embedding? The process and parameter for the embedding training was not described.\n\n4) In Section 4.3 \"batch size to eight\" better be consistent as \"batch size to 8\" (minor issue).\n\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "BkeK_KTlcH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "=== Summary ===\n\nThe authors propose a new abstractive dialog summarization dataset and task based on the MultiWOZ dataset. Unlike previous work which targets very short descriptions of dialog transcripts (e.g. 'industrial designer presentation'), this paper looks to generate long descriptions of the entire dialog using the prompts in the MultiWOZ task. The authors also extend the pointer generator network of See et al.  (2018) to use speaker, semantic slot and domain information.  They show that this new model (SPNet) outperforms the baseline on existing automatic metrics, on a new metric tuned to measure recall on slots (dubbed CIC), and a thorough human evaluation.\n\n=== Decision ===\n\nThe task of abstractive dialog summarization is well motivated and the field sorely needs new datasets to make progress on this task.  This paper is well written and executed, but unfortunately, I lean towards rejecting this paper because of a fundamental flaw in the nature of the proposed dataset that limits its applicability to the task of abstractive dialog summarization (more below).\n\nMy key concern is that the references in the dataset are generated from a small number of templates (Budzianowski et. al ,2018), which suggests this task is mostly one of slot detection and less about summarization.  The significant impact of including semantic slot information seems to be strong evidence this is the case. It is possible to rebut this concern with an analysis of how the generated summaries differ from the reference summaries. For example, Table 2 shows that sometimes the ordering of arguments is swapped: how often does this sort of behavior occur and how often do models identify information not in the reference?\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes an approach for abstractive summarization of multi-domain dialogs, called SPNet, that incrementally builds on previous approaches such as pointer-generator networks. SPNet also separately includes speaker role, slot and domain labels, and is evaluated against a new metric, Critical Information Completeness (CIC), to tackle issues with ROUGE. The reviewers suggested a set of issues, including the meaningfulness of the task, incremental nature of the work and lack of novelty, and consistency issues in the write up. Unfortunately authors did not respond to the reviewer comments. I suggest rejecting the paper.
Decision: Reject