Topic-Informed Dialogue Summarization using Topic Distribution and Prompt-based Modeling
Authors: Jaeah You
Youngjoong Ko
Venue: EMNLP 2023 Findings (2023)
Keywords: Dialogue Summarization
Topic Distribution
Topic-Informed Prompt
TL;DR: This paper presents TIDSum, a novel dialogue summarization model that considers topic distribution in the dialogue to generate the summary.

Abstract: Dealing with multiple topics should be considered an important issue in dialogue summarization, because dialogues, unlike documents, are prone to topic drift. Thus, we propose a new dialogue summarization model that reflects dialogue topic distribution to consider all topics present in the dialogue. First, the distribution of dialogue topics is estimated by an effective topic discovery model. Then topic-informed prompt transfers estimated topic distribution information to the output of encoder and decoder vectors. Finally, the topic extractor estimates the summary topic distribution from the output context vector of decoder to distinguish its difference from the dialogue topic distribution. To consider the proportion of each topic distribution appeared in the dialogue, the extractor is trained to reduce the difference between the distributions of the dialogue and the summary. The experimental results on SAMSum and DialogSum show that our model outperforms state-of-the-art methods on ROUGE scores. The human evaluation results also show that our framework well generates comprehensive summaries.

Topic-Informed Dialogue Summarization using Topic Distribution and Prompt-based Modeling
Jaeah You' and Youngjoong Ko²-
DDpaarmeent of Artificial Intelligence
Depaatment of Computer Science and Engineering Sungkyunkwan University youjaeahêgmail com, yjkoëskku. edu
Abstract
Dialgoue
(brief) Tina: How ahout pasm for dinner?
Sleve: Sounds great!
Dealing with multiple topics should be consid- ered important issue in dialogue summariza- tion, because dialogues, unlike documents, are prone topic drift. Thus, propose new dialogue summarization model that reflects di- alogue topic distribution to consider topics present in the dialogue. First, the distribution of dialogue topics is estimated by an effective topic discovery model. Then topic-informed prompt transfers estimated topic distribution in- formation to the output of encoder and decoder vectors. Finally, the topic extractor estimates the summary topic distribution from the out- put context vector of decoder to distinguish its difference from the dialogue topie distribu- tion. To consider the proportion of cach topic distribution appeared in the dialogue, the ex- tractor is trained to reduce the difference be- tween the distributions of the dialogue and the summary. The experimental results on SAM- Sum and Sua show that our model out- performs state-of-the-ee the methods on ROUGE scores. The human evaluation results also show that our framework well generates comprehen- sive summaries.
Tina: With broccoli, hum, cheese und cream?
Steve: Scrumptions.
Your favourite.
Steve: Indeed.
Tioa: But there snng
Steve: Too perfect
true? Tina: It's not about that. We'd need to do some shopping after work.
Can you handle it yoursel??
Steve: Can we handle it together? You know how scarterbruined am when comes to shopping lists.
Tina: do know!
Steve: Together?
Will you be leaving work on time!! C
Steve: Glusss so. don'tt expect any problems.
Tina: Let's meet in the car park, shall we?
Steve: Sure. ...(brigf)
BART generation
and work.
meet car park to some shopping after
TIDSam generation
Tina and will pasta dinnce. They will park and do the shopping after work together.
Figure I: Dialogue summary examples of SAMSum generated by BART and TIDSum.
Figure 1 shows an example of dialogue sum- mary generation results from BART (Lewis et al., 2019) and TIDSum models in the SAMSum dataset (Gliwa al., 2019); BART baseline model that has been widely used due to high performance in summary tasks. The content of the example dia- logue in Figure can be divided into three parts: Tina and Steve are having pasta for dinner, they will do the shopping together, and 3 they make an appointment to meet in the car park after Steve finishes work. That is, can think that the dia- logue contains three topics. However, BART did not capture the most important purpose of Tina and Steve's 's appointment to have pasta for dinner. They made an appointment to go shopping for ingredi- ents for pasta in dinner, so this should be included in the summary. Therefore, we focus on generat- ing more comprehensive summary that captures all the topics in the dialogue, without missing important topic.
1 Introduction
general, text summarization aims to generate a summary by capturing the core meaning from an original document consistently written by one par- ticipant on single topic. such news, scientific publications, etc (Rush et al., 2015; Nallapati et al., 2016). On the other hand, since dialogue consists of multi-speakers, the topic of the dialogue may be changed as topic drift according to the speaker's 's intentions (Zhao et al., 2020; Feng et al., 202 1)) a). Therefore, the dialogue summarization should take into account the distribution of multiple topics in a dialogue and reflect this distribution in generating the summary(Zou et al., 2021 a).
In this paper, we propose a novel model, Topic- Informed Dialogue Summarizer (TIDSum), that
Corresponding author

---

generates summary by considering the distribu- tion of topies existing in the dialogue. To estimate the distribution of multiple topics in a dialogue, we exploit the TopClus model to obtain the dis- tribution for the input dialogue, which was devel- oped for automatic topic discovery from text cor- pora (Meng et al.. 2022). Moreover, task-specific soft-prompll namely the topic-informed prompt, is added make the encoder context vector! well capture the dialogue topic information in the de- coding phase as well as the encoding phase. The topic-informed prompt is created by the latent em- bedding of the auto-encoder in the TopClus model. Since the latent embedding of dialogue from the TopClus model sufficiently contains the topic infor- mation of the dialogue, the topic-informed prompt influence the context vector of other tokens in the encoder and decoder of summarizer through the encoder self-attention and decoder cross-un auss-attention processes. The output context vectors from the de- coder are used for the topic extractor to estimate the topic distribution of the generated summary with it. Then the topic extractor provides an auxil- iary loss function to reduce the difference between the dialogue topic distribution and the summary topic distribution in the training phase. This learn- ing approach generates a better summary that well reflects the topic of the dialogue.
logue annotation tasks, including keywords extrac- tion, redundancy detection, and topic segmentation. Liu et al. (2021) proposed two topic-aware con- trastive learning objectives. This method implicitly modeled the topic change and handled information scattering challenges for the dialogue summariza- tion. Since summarizing dialogues is essential in customer services, Zou et al. (2021b) proposed topic-augmented two-stage summarizer with multi-role-spe coltiirole-ppeciiicc topic modeling mechanism. Li et al. (2022) presented a novel curriculum-basee prompt learning and applied a topic-aware prompt. from which we got the idea for a topic-informed prompt.
Topic-informed Summary Generation Framework
To perform the dialogue summarization effectively, necessary to identify the distribution of top- ics in the dialogue scattered across multiple utter- ances. Therefore, we propose a model to generate a topic-informed summary by reflecting the dialogue topic distribution to the summary topic distribution. The base architecture of TIDSSu is a Transformer- based regr-segressvve language model, BART.
3.1 Topic-Informed Prompt
In Figure 2-(1), we input dialogue into TopClus to obtain the latent topic embedding and dialogue topic distribution. To be specific, the latent topic embedding Itc is derived from the auto-encoder structure, TopClus, which ignores extrancous ele- ments and contains only salient information from the input. Each topic & is associated with the dialogue-topic distribution p(t/ltt), where is the number of topics². The distribution not only represents all the topical information present in the dialogue but also distinguishes between im- portant and unimportant topics. As you can see in Figure 2-(1), the topic-informed prompt tip is created by concatenating two Ites to match the dimensions of BART. Since Iee is a hidden state of the TopClus, an auto-encoder structure, it must be smaller than the input dimension of TopClus. The TopClus input is fixed at 768 dimensions be- cause it origins from the CLS token of BERT that encodes the input dialogue. Thus, set it to 512 dimensions to easily match the input dimensions of BART-large (1024 dims) by concatenating two
In the experiments, two daily dialogue summa- rization datasets, SAMSum and DialogSum, were used to evaluate our model. Compared to the previ- ous model, the proposed model improved the SOTA performance by 1.19%p and 1.94%p in Rouge-1 for SAMSum and DialogSum, respectively.
2 Related Work
Recently, there has been increasing attention on neural summarization for dialogues. Current stud- ies mainly apply transformer-based models (e.g.. BART (Lewis et al., 2019)) to abstractly summarize dialogues. However, these models are pre-trained on generic text corpora and essential fine- tune them a specific way for dialog data. Many studies have investigated how to find topics in di- alogues. Zhao et al. (2020) modeled the dialogue as an interactive graph according to the topic word information extracted from LDA (Blei et al., 2003). Feng et al. (2021b), which used DialoGPT (Zhang al., 2019) as the annotator, performed three dia-
'The last hidden states of the enender that are used as the decoder input are called the encoder context vector.
"We set the number of topics to 5 for SAMSum and 7 for Dua, which show the highest performance.

---

Encoder Topic Context Vector
Topic Distribution
Loss (Cepp)
Summary Topic Distribution
Encoder
Feed-forward Layer
Topic Extractor
Self-attention Layer
Decoder Topic-Informes Vector (dtt)
mcon pooling
Generation Loss (fggn))
Promet (ipi
Dialogue (D)
Decoder
Feed forward Layer
Giss
Latent Topic Embedding (ite)
Cross-attention Layer
TopClus
Dialogue Topic
Distribution
Masked-attention Layer
p(txlite)
(1)
(2)
Summary (?)
Figure 2: Overall framework of the proposed dialogue summarizer. (1)-() represent the order in which TIDSum operates.
Wes (1024 dims). Then, lip is located at the first token of the encoder input, preceding the dialogue D = [wi, 102, .... wn) with n tokens. The final in- put form is X = (tip, wI, u2; .... jn)- The topic information of the prompt tip infuses each token through a process known encoder self-attention during training. In this manner, the encoder con- text vectors are topically enhanced within the en- coder and their topic information is also propagated into the tokens of the decoder via encoder-decoder cross-attention.
where K is the number of topics. This ensures that the summary y is generated to reflect the dialogue topics.
3.3 Topic-informed Summary Generation
The generation loss Lyen is typically defined as the negative log-likelihood of the target summary given the input dialogue.
Eyen
Σ log - 1, (3) m==
The final loss & & is a weighted sum of the gen- cration loss Lyen and the topic distribution loss Ltopi
3.2 Topic Extractor
Figure 2-(2), the topic extractor, composed of MLP, serves to extract the distribution of the sum- mary. Through encoder-decoder attention, the topic information sourced from tip is reflected in the de- coder tokens. Therefore, we perform mean pooling on all tokens of the decoder to get the decoder topic-informed vector dti as follows:
Lfinal = Legnn ALtop
(4)
where À is a hyperparameter that controls the rel- ative importance these two losses. Experimen- tally, setting À to 0.75 showed the best performance. By minimizing the final loss f during training, the model is encouraged to generate summaries that are faithful to the input dialogue and well reflect the topic distribution of the dialogue.
1
dli M
Σ 9m
(1)
4 Experiments
where M is length of the summary and y {V1, 32. Fm } is the corresponding summary of
4.1 Datasets
tokens. The topic extractor estimates a summary topic distribution (dtt of k topics from dti. It is trained with cross-entropy loss to reduce the difference between the dialogue topic distribution and the summary topic distribution. The topic dis- tribution loss ftop is formulated as follows:
Two dialogue summarization datasets, SAMSum (Gliwa et al., 2019) and DialogSum (Chen et al., 2021), were used to verify the proposed model. SAMSum is an online chit-chat dataset from What- sApp and WeChat. DialogSum is real-life dia- logue dataset containing diverse task-oriented sce- narios and topics. It consists of a formal style of dialogue. Table shows the additional details for each dataset.
K
Lrop Σ
ptt lIte) log (ttl/dti)
(2)

---

Dataset
SAMSum
DialogSum
commensense knowledge. Li et al. (2022) ap- plies a curriculum-based prompt learning. Ctrl- DiaSumm-CoreffDA generates controllable sum- maries with personal named entity.
Language style
Written
Spoken
Domain
Online Chat
Daily life Dialogue
Train/Valid/Test
14,732/818/819
12,460/500/500
Avg. speakers
2.2
2
Avg. turns
8.4
13.8
4.4 Main Results
Avg. dia/sum length
94/25
131/23
To evaluate our model, we employed the ROUGE scores, which are widely used in summarization tasks. In detail, the Rouge-1, 1, Rouge-2 and Rouge- L variants, which consider unigram, bigram, and longest common subsequence overlap between gen- erated and reference summaries, were utilized in our experiments (Lin, 2004).
Table Data description for SAMSum and Dim,
Model
R-1
R-2
R-L
SAMSum
Transformer (Varaai al, 2017)
36.62
11.18
33.06
BART-Large ctsl, 2019)
53.06
28.18
49.50
TGDGA (Zae et 1000)
43.11
19.15
40.49
Table provides a comparison of our model with previous approaches on SAMSum and DialogSum.
BART(D A...) 2012
53.70
28.79
50.81
Ctrl-Dia DiaSummn+CrrffHtt
56.0
31.7
54.1
ReW ReWriteSum ('angea Pargeaal,am1
54,20
27.10
50.10
As shown in Table 2. TIDSum achieved the-state- of-the-art performances on both datasets. TIDSum
ConFiT (Tang ctal, 2022)
53.89
28.85
49.29
SICK++ + ct.. 2022)
53.24
28.10
48.90
obtains relative improvements of 1. 1.19%p Rouge-
(Li et al. 2022)
55.97
31.67
52.32
Our Architecture
1.71%p on Rouge-2 and 1.03%p on Rouge-L. compared with the previous SOTA model in SAM- Sum, and 1.94%p. 0.85%p and 3.07%p in Dialog- Sum. These results demonstrate the effectiveness of our technique for generating summary by dis- tinguishing and reflecting on the topics that appear in the dialogue.
1,
TIDSum
57.19
33,41
55.13
who Topic-informed prompt
56.77
32,90
54.99
wko Topic extractor
56.62
32.31
54.92
DinlogSum
Transformer (Vawari at, 2017:
35.91
8.74
33.50
Lagg (to. 2019)
45.19
19.69
43.08
RerriteSum (Farg etal.
35.1
14.6
32.1
SICK++ (km.
46.26
20.95
41.05
In the ablation study, the topic extractor found have the most impact on performance, as responsible for creating a summary topic dis- tribution in the framework that essentially reflects dialogue topics.
Our Architecture
TIDSum
48.20
21.80
46.15
who Topic-informed prompt
47.52
21.43
45.16
w/o Topic extractor
47.26
21.27
45.05
Table 2: ROUGE scores on the SAMSum from baseline models and proposed methods
4.5 Human Evaluation
4.2 Experimental Settings
Model
Info.
Cone.
Cov.
We loaded the pre-trained "facebook/barrtllagge"" for initialization. The learning rates of SAMSum and DialogSum were set to le-5 and 3e-5, and the train batch size were 2 and 4, respectively. The training was conducted at Nvidia Quadro RTX 8000 48G. We employed Py-rouge package to evaluate the models following (Feng al., 2021b; Liu and Chen, 2021).
Golden summary
3.83
4.19
3.70
BART-large
3.05
3.92
3.03
TIDSum who tip
3.26
3.99
3.30
TIDSum
3.69
3.97
3.71
Table 3: Human evaluation SAMSum. "Info." ". "Conc.", and "Cov." stand for Informativeness, Con- ciseness and coverage, respectively. w/o tip means that do not use the topic-informed prompt.
4.3 Comparison Models
For qualitative measurement of the generated summaries, we conducted human evaluations on three metrics by just following Feng et al. (202 1b). Informativeness evaluates how well the generated summaries capture more salient information. Con- ciseness measures how well the summary discards redundant information. Coverage measures how well the summary covers cach part of the dia- logue. We randomly sampled 60 dialogues with
TGDGA uses topic information and interactive graph structures. BART (D AD.) performs three dialogue annotation tasks using PLM. ReWrite- Sum used the utterance rewriting mechanism to complete the omitted content. ConFiT is also trained via a novel contrastive fine-tuning. SICK++ summarized the dialogue in way that utilizes

---

corresponding generated summaries to evaluate the SAMSum dataset. We asked three expert evalua- tors to rate each metric on scale of 5, with higher scores being better. The results are shown in Table 3.
logues (b) as shown in Figure 3. The result proves that TIDSum is effective for summarizing more multi-topic dialogues. In real-world scenarios, TID- Sum can be applied not only to simple dialogues between two speakers, but also to multi-party dia- logues, discussion summarization, etc. with more speakers and various topics. Figure 3 shows that the performance difference between TIDSum and BART-large is much larger in multi-topic than in single-topic. This suggests that applicable dialogues with more diverse topics.
In the case of informativeness, the golden sum- mary has the highest value because it is a summary written by person. However, TIDSum showed higher performance than BART. Conciseness is probably the shorter, the better, so was slightly higher for TIDSum using only the topic extractor than for TIDSum infused with more topic infor- mation, but the performance was almost the same. Finally, coverage is metric about whether sum mary covers the entire content of the dialog, and TIDSum scored higher than the golden summary this metric. This result shows that TIDSum effectively covers all the content of the dialogue.
Conclusion
We propose TIDSum, novel model for dialogue summarization. By reflecting the distribution of topies in the dialogue, TIDSum generates com- prehensive summaries. We utilize the TopClus model estimate topic distributions in the dia- logue, and introduce a task-specific soft-prompt. the topic-informed prompt, to capture and infuse topic information through the encoding and de- coding phases. The generated summaries were evaluated using SAMSuu and DialogSum datasets, and our model outperformed previous approaches with a significant improvement in ROUGE scores and human evaluation results. Overall, TIDSum effectively captures and summarizes the details of each topic in the dialogue, resulting in high-quality summaries.
5
Analysis
60
BART-large
TIDSum
50
45
35
Limitations
30
The proposed method needs train the TopClus model with the dialogue data to get the topic distri- bution and latent topic embedding of the dialogue before fine-tuning the BART based summarization model. Since TopClus is an auto-encoder model with high dimensional layers, it takes long time to train. With the obtained topic distribution and latent topic embedding, the BART based summa- rization model is trained and generates summary in inference phase. This two-stage process is com- plicated and requires some time. Therefore, order simplify this process, our future work to incorporate only the essential parts of TopClus into the main lcarning process.
25
R-1
R-2
R-L
R-1
R-2 (b) Multi-Topic
(a) Single-Topic
Figure 3: ROUGE scores differences for single-topic and multi-topic dialogues
Comparison for Single-Topic vs Multi- Dialogues
We herein attempt to verify that our model works better in multi-topic dialogues than in single-topic dialogues and it can generate comprehensive sum- maries well. The SAMSum test dataset was divided into single-topic and multi-topic ones. Dialogues with an entropy value of topic distribution less than 0.5 were regarded single-topic ones. Eventually, the total dialogues are separated into 178 single- topic dialogues and 641 multi-topic dialogues. result, TIDSum showed larger improvement differ- ences over baseline, BART-large multi-topic dia-
A Akknowledgements
This work was supported in part by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (No. NRF- 2020R I 12C220) in part by Institute for Infor-

---

mation & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No. 2022-0-00369, (Part 4) Development of AT Technology to support Ex- pert mecisinn that can Explain the Rea- sons/Grounds for Judgment Results based on Ex- pert Knowledge), in part by Institute of Information & Communications Technology Planning & Evalu- ation (IITP) grant funded by the Korea Government (MSIT) (No. 2020-0-00368, A Neural-Symboll Model for Knowledge Acquisition and Inference Techniques), and in part by Institute of Information & communications Technology Planning & Evalu- ation (IITP) grant funded by the Korea government (MSIT) (No.2019-0-00421, AT Graduate School
Mike Lewis. Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy. Ves Stoyanov, and Luke Zetlemoyer. 2019. Bart: De- noising seguence-to-sqquene pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv: (919191116666
Changqun Li, Linlin Wang, Xin Lin, Gerard de Melo. and Liang He. 2022. Curriculum prompt learning with self-training for abstractive dialogue summa- rization. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 1096-1 106, Abu Dhabi, United Arah Emirates. Association for Computational Linguistics.
Chin-Yerr Yew Lin. 2004. Rouge:
A package for automatic evaluation of summaries. In Text summarization branckes out, pages 74-81.
Support Program (Sungkyunkwan University))
Junpeng Liu, Yanyan Zou, Hainan Zhang, Hongshen Chen, Zhuoye Ding, Caixia Yuan, and Xiaojic Wang. 2021. Topic-aware contrastive learning for abstrac- tive dialogue summarization. In Findings of the Asso- ciation for Computational Linguistics: EMNLP 2021. pages 1229-12-44 1243, Punta Cana. Dominican Republic. Association for Computational Linguisties.
References
David M Blei, Andrew Ng. and Michael Jordan. 2003. Latent dirichlet allocation, Journal of machine Learning research, 993-
Zhengyuan Liu and Nancy Chen. 2021. Controllable neural dialogue summarization with personal named entity planning. In Proceedings of the 2021 Con- ference on Empirical Methods in Natural Language Processing, pages 92-106, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Yulong Chen, Yang Liu, Liang Chen, and Yue Zhang. 2021. DialogSum: A real-life scenario dialogue sum- marization dataset. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages
Online. Association for Computa- tional Linguistics.
Yu Meng, Yunyi Zhang. Jiaxin Huang, Yu Zhang, and Jiawei Han. 2022. Topic discovery via latent space clustering of pretrained language model representa- tions. In The Web Conference.
Yue Fang. Hainan Zhang. Hongshen Chen, Zhuoye Ding, Bo Long, Yanyan Lan, and Yanquan Zhou. 2022. From spoken dialogue to formal summary: An utterance rewriting for dialogue summarization. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, pages 3369.
Ramesh Nallapati, Bowen Zhou, Caglar Gulcehre, Bing Xiang, et al. 2016. Abstractive text summarization using preprint
rmss and beyond. arXiv
Alexander M Rush, Sumit Chopra, and Jason We- ston. 2015. A neural attention model for ab- stractive sentence summarization. arXiv preprint 1cXi::1509.00683
Xiachong Feng, Xiaocheng Feng, and Bing Qin. 2021a. A survey on dialogue summarization: Re- cent advances and new frontiers. arXiv preprint 11Xiv:2107.03175.
Xiangru Tang, Arjun Nair, Borui Wang. Bingyao Wang. Jai Desai, Aaron Wade, Haoran Li, Asli Celikyil- maz, Yashar Mehdad, and Dragomir Radev. 2022. CONFIT: Toward faithful dialogue summarization with linguistically-infon -nnfoomme contrastive fine-tuning. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies. pages 5657-5668, Seattle, United States. Association for Computational Linguistics.
Xiachong Feng, Xiaocheng Feng, Libo Qin, Bing Qin, and Ting Liu. 202 Language model as an annota- tor: Exploring dialogpt for dialogue summarization. arXiv preprint arXiv:2105.... 12544.
Bogdan Gliwa, Iwona Mochol, Maciej Biesek, and Alek- sander Wawer. 2019. Samsum corpus: A human- annotated dialogue dataset for abstractive summa- rization. arXiv preprini
Seungone Kim, Se June Joo, Hyungjoo Chae, Chae- hycong Kim, Seung-won Hwang, and Jinyoung Yeo. 2022. Mind the gap! injecting commonsense knowl- edge for abstractive dialogue summarization. arXiy preprint 2rXiv:2209.00933
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances neural information processing systems, 30.

---

Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen. Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing Liu, and Bill Dolan. 2019. Dialogpt: Large-scale generative pre-training for conversational response generation. arXiv preprini arXiv: 1911v::911.0033 90110100060
Lulu Zhao, Weiran Xu, and Jun Guo. 2020. Improving ahstractive dialogue summarization with graph struc- tures and topic words. In Proceedings of the 28th International Conference on Computational Linguis- rics, pages 437-449.
Yicheng Zou, Lujun Zhao. Yangyang Kang, Jun Lin, Minlong Peng, Zhuoren Jiang, Changlong Sun, Qi 7hang. Xuanjing Huang, and Xiaozhong Liu. 2021a. Topic-oriented spoken dialogue summariza- tion for customer service with saliency-awe aware topic modeling. Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 14665- 14673.
Yicheng Zou, Lujun Zhao, Yangyang Kang, Jun Lin, Minlong Peng, Zhuoren Jiang, Changlong Sun, Qi Zhang, Xuanjing Huang, and Xiaozhong Liu. 2021b. Topic-oriented spoken dialogue summariza- tion for customer service with saliency-aware topic modeling. Proceedings of the AAAI Conference Artificial Intelligence. 35(16):14665 146734


Reviews:
[{"review_id": "aYXwScZ972", "reviewer": "Reviewer_GoNj", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> **Q1.** The author's model description is not clear. For example, in line 149, I don't quite understand how the $tip$ is calculated, please explain further here or use a formula example to illustrate. \n\n**A1.** $tip$ (1024 dims) is a concatenation of two $lte$ (512 dims). Since $lte$ is a hidden state of the TopClus autoencoder, it must be smaller than TopClus autoencoder input dimension (768 dims). The TopClus input is fixed at 768 dimensions because it origins from the CLS token of BERT that encodes the input dialogue. Thus, we set it to 512 dimensions to easily match the input dimensions of BART-large (1024 dims) by concatenating two $lte$ (1024 dimensions) as follows:\n\n$$tip = lte \\oplus lte$$\n\nwhere $\\oplus$ means concatenation.\n\n> **Q2.** Insufficient experiments. Only ablation experiments and main experiments are shown, lack of model analysis experiments and sample analysis.\n\n**A2.** We conducted several additional analytical experiments, but for the limited space of the short paper, only the core experiments were presented in the main text. Appendix A.4 shows an evidence that our proposed model performs better in summarizing dialogues with multiple topics. When we evaluated the generation results by dividing the dialogues into single-topic and multi-topic dialogues, the performance difference between BART-large (baseline) and TIDSum was much larger for multi-topic dialogues. It is an important evidence that our model works better in complex dialogues with multiple topics (e.g., multi-party dialogues) than in relatively simple and short dialogues with a single topic. \n\nAs shown in Table 1, we also found that the average attention weight of topic-informed prompt ($tip$) are relatively higher than the other tokens, which means that the topic information in $tip$ is more concentrated in the sequence, allowing the model to generate more meaningful summaries. \n\n|Average attention weight of $tip$|Average attention weight of the other tokens|\n|:---:|:---:|\n|**0.3044**|0.0197|\n\n**Table 1. Comparison of the average attention weight of $tip$ to the average attention weight of the other tokens, for 50 examples**", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1)The proposed Topic-Informed Dialogue Summarizer (TIDSum) explicitly considers the distribution of multiple topics within a dialogue. This approach allows the generated summaries to comprehensively capture all the important topics present in the dialogue, resulting in summaries that are more comprehensive and representative of the original conversation.\n2)The paper introduces the concept of a \"topic-informed prompt\" that is integrated into the summarization model's encoding and decoding processes. This prompt is derived from the latent embedding of the TopClus model, which estimates the distribution of dialogue topics. By leveraging this prompt, the model enhances the context vectors of both the encoder and decoder, ensuring that topic information is effectively captured throughout the summarization process. \n3)The experimental results presented in the paper indicate that the proposed TIDSum model outperforms existing methods on two daily dialogue summarization datasets, SAMSum and DialogSum.", "reasons_to_reject": "1）The author's model description is not clear. For example, in line 149, I don't quite understand how the tip is calculated, please explain further here or use a formula example to illustrate\n2）Insufficient experiments. Only ablation experiments and main experiments are shown, lack of model analysis experiments and sample analysis.", "questions_to_authors": "", "paper_topic_and_main_contributions": "Since the context of dialogues with multiple topics prone to topic drift, the authors propose a novel dialogue summarization model called Topic-Informed Dialogue Summarizer (TIDSum), which aims to capture and reflect all the topics present in dialogues. They utilize an effective topic discovery model, called TopClus, to estimate the distribution of multiple topics within a dialogue. Then they conduct a task-specific soft-prompt called the \"topic-informed prompt\", which is derived from the latent embedding of the dialogue obtained from the TopClus model. Finally, by employing the topic-informed prompt, the model influences the context vectors of the encoder and decoder, ensuring that the topic information is considered throughout both encoding and decoding phases of summarization.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "5ijpgcrahE", "reviewer": "Reviewer_KN1n", "summary": "", "questions": "A. I am wondering how you train the topic extractor.  Since latent topic embedding lte is derived from the TopClus and dti is the mean pooling of all tokens, they are projected in different semantic embedding spaces. Do you have one extra step besides summary training for the topic extractor?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> **Q1.** The paper selected \"number of topics to 5 for SAMSum and 7 for DialogSum\". It seems the dataset contains very limited topics which is an uncommon scenario. Given the relatively short length of these two datasets, one may question the effectiveness of the proposed method when applied to dialogues with a more extensive range of topics and longer length. \n> **Q2.** While dialogue contains multiple topics, the summary of dialogue doesn't necessarily need to cover all the topics mentioned in the dialogue. Can you test the topic similarity of the gold standard summary and dialogue inputs to see what's the upper bound?\n\n**A1 & A2.** We'll write the responses to Q1 and Q2 together. We have merely found the optimal number of topics that can appear in the entire dataset to get the highest performance. If a wider range of topics and longer dialogues are to be covered, a larger number of topics can be specified and trained on. Note that setting the number of topics to five does not ensure that five topics will appear, but rather that this is the maximum number of topics that can appear. We sampled 50 examples from the SAMSum dataset and counted the number of topics with probability ≥ 0.1. The dialogue topic distribution was dominated by about 2.1 topics, and the golden summary topic distribution was dominated by about 1.7 topics. Table 1 is an example of the specific distributions. Even if a larger dataset is used and 11 topics are designated, there will still be dominant topics in any given dialogue. We agree that you don't need to cover every topic in the dialogue to generate the summary. However, our approach allows the summary to better reflect a few of the more influential topics in the dialogue.\n\n|Index|Dialogue Topic Distribution|Summary Topic Distribution|\n|:---:|---|---|\n|1|[0.0004, 0.0010, 0.0003, 0.2984, 0.6999]|[0.0002, 0.0016, 0.0003, 0.4693, 0.5285]|\n|2|[0.0018, 0.1442, 0.0014, 0.4161, 0.4365]|[0.0009, 0.1376, 0.0009, 0.0856, 0.7750]|\n|3|[0.6036, 0.0012, 0.2248, 0.0011, 0.1693]|[0.6178, 0.0002, 0.3782, 0.0001, 0.0037]|\n|4|[0.0018, 0.9308, 0.0001, 0.0671, 0.0002]|[0.0007, 0.9143, 0.0012, 0.0002, 0.0836]|\n\n**Table 1. Samples of the dialogue topic distribution and the golden summary topic distribution in the SAMSum dataset**\n\n\n> **Q3.** I am wondering how you train the topic extractor. Since latent topic embedding $lte$ is derived from the TopClus and $dti$ is the mean pooling of all tokens, they are projected in different semantic embedding spaces. Do you have one extra step besides summary training for the topic extractor?\n\n**A3.** $lte$ and $dti$ are not in different semantic embedding spaces. Two $lte$ concatenate to become a $tip$, which gives topic information to other tokens through attention, and also influences the generation of $dti$. The dialogue topic distribution $p(t_k|lte)$ extracted from TopClus is frozen distribution because it has been extracted previously. A single MLP, topic extractor, extracts the summary topic distribution from $dti$ and is trained to make the summary topic distribution similar to the pre-extracted dialogue topic distribution with cross entropy loss. In other words, topic extractor is trained to make the summary topic distribution equal to the dialogue topic distribution. As the topic extractor is trained to get better at extracting the summary topic distribution, the decoder is trained to generate summaries better.\n\n\n> **Q4.** For the human evaluation part, can you give a formula definition of what's conciseness and coverage?\n\n**A4.** The metrics used for human evaluation just followed the approach of _Feng et al. 2021b_. Among these metrics, conciseness is a metric that measures the extent to which the summary discards redundant information. The number of incorrectly generated sentences or unnecessary sentences $k$ for all $l$ sentences is calculated by the following formula:\n$$Round(5*(1-\\frac{k}{l}))$$\n\nCoverage is a measure of how well the summary reflects all the key topics that appeared in the dialogue. If there are $n$ topics of dialogue that the evaluators think should be included in the summary, and the generated summary contains $m$ $(m≤n)$ of them, the formula is as follows: \n$$Round(5*\\frac{m}{n})$$", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper is well written and the model achieves the SOTA performance in SAMSum and DialogSum datasets.\n", "reasons_to_reject": "1. The paper selected \"number of topics to 5 for SAMSum and 7 for DialogSum\". It seems the dataset contains very limited topics which is an uncommon scenario. Given the relatively short length of these two datasets, one may question the effectiveness of the proposed method when applied to dialogues with a more extensive range of topics and longer length.\n2. while dialogue contains multiple topics, the summary of dialogue doesn't necessarily need to cover all the topics mentioned in the dialogue. Can you test the topic similarity of the gold standard summary and dialogue inputs to see what's the upper bound? ", "questions_to_authors": "A. I am wondering how you train the topic extractor.  Since latent topic embedding lte is derived from the TopClus and dti is the mean pooling of all tokens, they are projected in different semantic embedding spaces. Do you have one extra step besides summary training for the topic extractor?", "paper_topic_and_main_contributions": "The paper proposes a new dialogue summarization model that minimizes the gap between summary and dialogue topic distribution. And the model achieves the SOTA performance in SAMSum and DialogSum datasets.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. For the human evaluation part, can you give a formula definition of what's conciseness and coverage?\n2. Citation format L237"}, "extra_text": {}}, {"review_id": "OM8e5uz6Vq", "reviewer": "Reviewer_Fkky", "summary": "", "questions": "Can you provide more details on the human evaluations conducted in the experiments? How reliable are these evaluations, and are there any potential biases or limitations in the evaluation methodology? ", "limitations": "", "rating": 3, "confidence": 2, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> **Q1.** The paper does not discuss the scalability of the proposed model, which could be a potential limitation in real-world scenarios. A discussion of the model's scalability and potential challenges in scaling it up could help readers better understand the practical implications of the proposed model.\n\n**A1.** In real-world scenarios, the proposed dialogue summarization model can be applied not only to simple dialogues between two speakers, but also to multi-party dialogues, discussion summarization, etc. with more speakers and various topics. Appendix A.4 shows that the performance difference between TIDSum and BART-large is much larger in multi-topic than in single-topic. This suggests that it is applicable to dialogues with more diverse topics. \n\n> **Q2.** Can you provide more details on the human evaluations conducted in the experiments? How reliable are these evaluations, and are there any potential biases or limitations in the evaluation methodology?\n\n**A2.** Human evaluations were conducted by three linguistic experts. For both the golden summary and the output summary of all models, the evaluators were blinded to which model produced the output. There are no potential biases or limitations because the evaluators assessed the randomly mixed summaries without knowing which model the output was for. To assess inter-annotator agreement, we measured the Fleiss' kappa coefficient and achieved scores of 0.43, 0.35, and 0.40 for informativeness, conciseness, and coverage, respectively.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper proposes a new model for Topic-Informed Dialogue Summarization that considers all topics present in a dialogue, reflecting dialogue topic distribution. This model is sound.\n2. The model outperforms baselines on ROUGE scores.", "reasons_to_reject": "The paper does not discuss the scalability of the proposed model, which could be a potential limitation in real-world scenarios. A discussion of the model's scalability and potential challenges in scaling it up could help readers better understand the practical implications of the proposed model.", "questions_to_authors": "Can you provide more details on the human evaluations conducted in the experiments? How reliable are these evaluations, and are there any potential biases or limitations in the evaluation methodology? ", "paper_topic_and_main_contributions": "This paper proposed a new model for Topic-Informed Dialogue Summarization that considers all topics present in a dialogue, reflecting dialogue topic distribution. The model estimates the distribution of dialogue topics and generates comprehensive summaries. The model outperforms baselines on ROUGE scores.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper introduces topic-informed dialogue summarisation, which pre-computes the topic distribution in a dialogue and incorporates this information (in the form of soft prompt / latent embeddings) in the summarisation process. Reviewers thought the idea of incorporating topic distribution, particularly for dialogues, is well-motivated and the proposed approach is interesting. Moreover, reviewers also thought quantitatively the topic-informed summariser appears to perform very well. That said, there are some minor but important concerns that dampens the excitement of reviewers: (1) the writing for certain parts in the method description can be improved (GoNj and KN1n); (2) the paper would benefit from more analyses to bring out more insights (GoNj); (3) the utility/applicability of the approach for longer conversations warrants at least a discussion, if not more results (KN1n, Fkky); and (4) the paper might want to provide some results to validate the assumption that the topic distribution in the ground truth summary is similar to that of the dialogue.
Decision: Accept-Findings