Instructive Dialogue Summarization with Query Aggregations
Authors: Bin Wang
Zhengyuan Liu
Nancy F. Chen
Venue: EMNLP 2023 Main (2023)
Keywords: Dialogue Summarization
Query-based Dialogue Summarizaiton
Dialogue Reading Comprehension
TL;DR: We present instructive dialogue summarization (InstructDS) model which can incoporate user queries when generating dialogue summaries.

Abstract: Conventional dialogue summarization methods directly generate summaries and do not consider user's specific interests. This poses challenges in cases where the users are more focused on particular topics or aspects. With the advancement of instruction-finetuned language models, we introduce instruction-tuning to dialogues to expand the capability set of dialogue summarization models. To overcome the scarcity of instructive dialogue summarization data, we propose a three-step approach to synthesize high-quality query-based summarization triples. This process involves summary-anchored query generation, query filtering and query-based summary generation. By training a unified model called InstructDS (Instructive Dialogue Summarization) on three summarization datasets with multi-purpose instructive triples, we expand the capability of dialogue summarization models. We evaluate our method on four datasets, including dialogue summarization and dialogue reading comprehension. Experimental results show that our approach outperforms the state-of-the-art models and even models with larger sizes. Additionally, our model exhibits higher generalizability and faithfulness, as confirmed by human subjective evaluations.

Instructive Dialogue Summarization with Query A Aggregations
Bin Wang, Zhengyuan Liu, Nancy F. Chen
Institute for Infocomm Research (I²R), A A*STAR, Singapore wang_binei2r wereeiiin...eee a-s.aa.ssee..... edu. sg
Abstract
Benjamin: Hey guys, what we doing with the keys today?
Hilary: I've got them. Whoever wants them can meet me lunchtime after
Conventional dialogue summarization methods directly generate summaries and do not con- sider user's specific interests. This poses chal- lenges in cases where the users are more fo- cused on particular topics or aspects. With the advancement of instructioo-finettuedd lan- finetuned
Elliot: I'm ok. We're meeting for the drinks in the evening anyway and
Benjamin:
Interesting @ @e honest, Hilary, almust feel like changing my mind. Wanting to take this nap might end costing me dear
guage models, we introduce instruction -tuning dialogues to expand the capability set of dia- logue summarization models. To overcome the scarcity of instructive dialogue summarization data, we propose a threc-step approach syn- thesize high-quality query-based summariza- tion triples. This process involves summary- anchored query generation, query filtering and query-based summary generation, By training a unified model called InstructDS (Instructive Dialogue Summarization) on three summa- rization datasets with multi-purpose instruc- tive triples, we expand the capability of dia- logue summarization models. We evaluate our method on four datasets, including dialogue summarization and dialogue reading compre- hension. Experimental results show that our ap- proach outperforms the sfate-of-the-art models and even models with larger sizes. Additionally, our model exhibits higher generalizability and faithfulness, as confirmed by human subjective evaluations.
Hilary: Do join we're going have fun. And then you'll take the keys and take this most deserved of naps
Elliot:
Sounds like plan
Hilary:
Elliot: See you then
Instruction:
Where is Hilary having lunch?
Output: La Cantina
Highlight the key takeaways from the dialogue. Output: Hilary has keys the apartment. Benjamin wants get them and go take nap. Hilary is having lunch with some French people
Instruction:
Cantina. Hilary ...
Summarize the dialogue with about 15 words. Output: Benjamin, Hilary and Elliot discussing about their plans for the day after getting the apartment keys.
Instruction:
Figure 1:
Instructive dialogue summarization models, such as InstructDS, demonstrate multiple capabilities.
dialogue-summary summary pairs as straightforward input- output pairs (Shang et al., 2018; Goo and Chen, 2018; Chen and Yang, 2020). Although fine-tuning pre-trained language models such as BART (Lewis et al., 2020) has shown promising results, these models fail to consider the specific preferences of users who have distinct backgrounds, objectives. intents, and applications for the summaries they require.
1 Introduction
Both Verbal and non-verbal conversations play crucial role in the realm of communication. They serve as channels for humans to exchange infor- mation, ideas and emotions (Kester, 2004). In an era of information explosion overload, dialogue summarization has become increasingly essential. The process involves extracting the key dialogue information, enabling people to grasp the essence of their daily interactions.
In order to address this challenge, several meth- ods have been proposed to integrate queries when generating summaries (Dang, 2006; Nema et al., 2017; Su et al., 2021; Zhong et al., 2021; Zhu et al., 2022; He et al., 2022). However, these models primarily concentrate on domains such news (Dang, 2006; He al., 2022), Wikipedia (Zhu et al., 2022), and meetings (Zhong et al., 2021).
Conventional dialogue summarization mod- cls typically approach the problem as an un- task, treating
constrained
The exploration of based summarization for

---

dialogues remains limited. Furthermore, Liu and Chen (2021) propose controllable generation using personal named entity planning, and Wang et al. (2022a) suggest controlling summary conciseness. However, both methods focus on specific aspects of controllability and still lack the flexibility to in- corporate user requirements as shown in Figure 1.
triples from diumaay pairs, facil- itating query-based dialogue summarization. This method leverages the question generation and answering capabilities of large language models (LLMss). We validate its effectiveness through evaluations conducted by human an- notators.
The primary obstacle in instruction-basse dia- logue summarization is the scarcity of training data. While existing datasets contain dialogue-summayy pairs, creating query-base dialogue summariza- tion datasets with limited human involvement is challenging due to high costs, limited diversity, and potential quality issues. In this work, shed light by Self-Instruce (Wang et al., 2022c), propose to synthesize query-dialogue-summass by leveraging the conditional question generation and answering ability of general large language models (LLMs) (Wei et al., 2023). The process involves requesting LLMs to generate multiple can- didate queries based on the reference summary. A filtering mechanism, employing text-based and semantic-based methods, is then applied to ensure the quality of collected queries. Finally, the query- based summarization generated by triggering the question -enestion-nnswering ability of LLMs.. This ap- proach demonstrates promising solution to gen- erate query- query-based -ased dialogue summarization triples while reducing human involvement and enhancing data diversity.
We conducted an extensive evaluation on 3 di- alogue summarization datasets and 1 dialogue comprehension dataset. The results demon- strate substantial improvement models. Additionally, according to human subjective test, our generated summaries ex- hibit comparable levels of factuality, fluency, informativeness, and conciseness to human written ones.
previ-
(QDS) triples
Related Work
2.1 Dialogue Summarization
Dialogue summarization is the task of generat- ing a concise and fluent summary of conversa- tion involving two or more participants. It has gained significant attention due to its broad appli- cations and availability of relevant datasets (Gliwa et al., 2019; Chen et al., 2021; Zhao et al., 2021). Solutions on dialogue summarization are mainly based on setpueencc----equeene models including the pointer-generation network (See al., 2017). T5 (Raffel et al., 2020) and BART (Lewis et al., 2020). However, it remains challenging task due to the lengthy and unstructured nature of dia- logues. Chen and Yang (2020) proposes extracting dialogue structures from various perspectives be- fore summarization. Other approaches attempt to incorporate co-reference information (Liu et al., 2021b) and leverage dialogue understanding objec- tives (Liu et al., 2021a) to enhance the factuality and informativeness (Tang et al., 2022; Wang al., 2022b).
The InstructDS framework is shown in Figure 2. Through joint training with QDS triples, InstructDS can cater to user preferences by producing query- based summarics. This mixed training paradigm enhances the model's understanding of dialogue, which improves the factual quality of generated summarics. Our model exhibits superior domain generalizability by incorporating multiple datasets into unified framework and the user's conciseness requirement can be fulfilled by length-aware augmentations.
Similar to text summarization, the process of generating dialogue summarization is uncontrol- lable and poses challenges in incorporating user preferences (Zhong et al., 2021; He et al., 2022). Efforts have been made to enhance the controlla- bility of dialogue summarization. However, these approaches often have limited focus on personal named entities (Liu and Chen, 2021, 2022: Wang et al., 2022a) and conciseness (Wang et al., 2022a). The primary challenge in instructive dialogue sum- marization lies the availability of suitable su-
Our main contributions are summarized as fol- lows:
We introduce InstructDS, the pioneering cortuuctioo-oolowinng dialogue summarization model. It is text generation model designed to summarize dialogues while explicitly con- sidering user instructions.
We present a straightforward yet effective ap- proach to synthesize query- saary-dillouu--ummmee

---

Ganeral Summariaaion Dabase
pasa
Detazet A
Shard poing to to the
Diologue
BB What is Bivaron afradd of?
para
pass
Geremnton
General-Purpose LLM
oa
dosse lik hee??
Summary
p4: Who Sharen going the beach
pasd
Datenet B
os Whot Shardfo relodionsiio with Wordly
fail
Dalogue
2. Aitaring and Pastprecesing
Saand
Dialague
cased El Carque Senmartattor
Summary
InstructDS
Instruction Dataset
Sucte seveler
Dalogue
Summary
Ole Instackann parded
Figure Overall framework of our Instructive Dialogue Summarization (InstructDS) model.
pervision. While QMSum (Zhong et al., 2021) introduces the first query-based meeting summa- rization, it focuses on lengthy meetings and con- sists of only 232 meeting samples. To address this limitation, we propose a methodology for synthe- sizing query-dialogue-summaal triples leveraging summary- anchored techniques, to facilitate instruc- tive dialogue summarization.
the summary is dependent on both query and di- alogue. In non-instructive dialogue summariza- tion, given dialogue-summan pair (Dui, Sd,; ) from dataset d with pa 2 pairs of instances, model M is expected generate summary given the corresponding dialogue: M(Da,))
Sā.5- In instructive dialogue summarization, the focus shifts to structured triples. Given a query- dialogue summary (QDS) triple [Qu, Da, Sa) from dataset d with to triples, an instructive model M should generate summary conditioned on both query and dialogue: (QQ,, Da,)) Sa, Note that the same dialogue be shared in several QDS triples with different queries.
2.2 Instruction Tuning
Recently, instruction -inetttiinnnnn on large language models has demonstrated remarkable generalizabil- ity towards unseen tasks by leveraging task de- scriptions (Brown et al., 2020; Wang et al., 2022d; Chung et al., 2022). The availability of high-quality and diverse instructions unlocks the emerging capa- bilities of LLMs. For instance, Flan-series models are tuned with over 1800 tasks with diverse instruc- tions (Chung et al., 2022). However, dialogue tasks, being a sub-domain, have limited supervised data. This limitation leads to sub-optimal performance on query-based dialogue summarization using ex- isting -inetuned models. To mitigate the reliance on Self-Instruct (Wang et al., 2022c) uses GPT3 for generating diverse instructions and input-output pairs. similar vein, our study introduces diverse and high-quality augmentations of query-based di- alogue summarization data for instructive dialogue summarization.
3.2 Synthesize QDS Triples
The process of generating query query-dialogue- summary (QDS) triples from dialogue-summary pairs in pipeline involves three steps: 1) query generation from complete summary; 2) query filtering to ensure the validity and diversity; 3) query-gun query-guided summary generation.
Query Generation. In order to capture a di- verse range of potential queries, we deploy the question-generannn
instruction data,
ability of LLMs to generate multiple candidate queries. Specifically, Flan-T5-XL (Chung et al., 2022) (refer as model X) which has been trained on question generation datasets such as Quoref (Dasigi et al., 2019), MC- TACO (Zhou et al., 2019) and CosmosQA (Huang et al., 2019). We expect that its question--enerattoo ability can be generalized to other narrative text. For each instance, queries using the template shown in Table 7.
3 Instructive Dialogue Summarization
3.1 Problem Definition
generate five candidate
Unlike previous dialogue summarization ap- proaches, instructive dialogue summarization in- volves a controllable generation process where
Filtering and Postprocessing. To ensure va- lidity and diversity, we present two methods for

---

Direct Exposure
Dataset
# Train
Validation
Test
# QDS Triples
Alpaca
Flan-Series
InstructDS
SAMSum (Gliwa et al., 2019)
14,732
818
819
18,245
DialogSum (Chen ul., 2021)
12,460
500
1,500
18,600
TODSum (Zhao ct al., 2021)
7,892
999
999
8,705
x
X
/
DREAM (Sun ul., 2019)
6,116
2,040
2,041
X
X
The dataset statistics include several dialogue summarization datasets such SAMSum, DialogSum, and TODSum, well as the DREAM dataset, which focuses on dialogue reading comprehension and contains natural wwwer-iaalolummaaan triples. The right part indicates the direct supervision exposures for Alpaca, Flan-Series, and InstructDS models.
Table I:
Quality Review Question
Yes% (wfo filtering)
more concise and structured source, which further guarantees the quality.
Does the query question answerable? Is the query di differs from previous ones for the same dialogue? Is the generated sunnary correct and acceptable for query and dialogue? Both unique and correct.
94% (76%)
90% (63%)
Quality Check. Finally, we collect QDS triples for three dialogue summarization datasets and present statistics in Table 1. On average, 1.3 QDS triples are generated for each dialogue-summary pair. To access quality and diversity, we enlist help from an expert to annotate 100 triples. Evalua- tion results in Table 2 demonstrate a significant im- provement in the quality of synthesized triples after applying our filtering technique, with the quality score increasing from 45% to 75%. In the process, incorporate the summary information as it rep- resents condense version of the information con- tained in the corresponding dialogues. The triples gathered are tend to have higher quality with fewer errors and cover more utterances as the comparison shown in Table 10.
83% (71%)
75% (45%)
Table 2: Quality review for the generated queries and summaries from synthesized DQS triples. After (before) filtering results shown. Examples of both kept and filtered QDS triples can be found in Table 10, 11. 12.
query filtering. 1) Text-based filtering. Through an analysis candidate queries, we observe that some queries are not answerable conclusively with- out hallucinations. Examples of such queries in- clude 'What will' and 'How would' queries. There- fore, we utilize model X as a text-based binary classifier to determine the answerability of queries using the template in Table 7. This filtering pro- cess eliminates around 45% of generated queries that are likely to be unanswerable. 2) Semantic- based filtering. avoid redundancy and ensure diversity, we remove similar queries for the same dialogue-summary pair through semantic similarity measurement. For instance, queries such as 'What does Edward think about Bella?' and 'What does Edward think of Bella?' are almost identical in meaning. We keep only the first query if the seman- tie similarity score, computed using normalized BERTScore (Zhang al., 2020), is above 0.65. The based filtering process eliminates an additional 50% of the queries.
3.3 Model Training
We perform instruction tuning with Flan-T5-XL model as the initial checkpoint. The instructions are three-fold: 1) general dialogue summariza- tion, 2) query-based dialogue summarization and 3) their length-aware augmentations. For query- based dialogue summarization, the query and dia- logue are concatenated as the input with the tem- plate: "###Insttuctio:: (instruction). ### Input: [dialogueJ.", where output is the summary. To account for length-aware generations, we append "The generated summary should be around /sum- mary length) words long." " to the original instruc- tion.
Query-based Summary Generation. Using the query and the complete summary as input, we gen- crate the query-based summary with model X. It is worth noting that generating query-based sum- maries from dialogues challenging for model X. In contrast, generating query-based summaries from condensed summarics is compuratively casier as allows the model to extract information from a
To enhance the generalizability across different dialogue types, we combine three dialogue summa- rization datasets to train unified dialogue summa- rization model. From the synthesized QDS triples. we random sample 5k triples from each dataset. For length awareness, each sample is augmented once. This results total of (14.7k: + 12.5k +

---

ROUGE-1
ROUGE-2
ROUGE-L
Models
Params
BS
Fi
Pre
Rec
Fi
Pre
Rec
Fi
Pre
Rec
Dointer-Generaie
40.1
15.3
36.6
BART
400M
53.0
59.0
52.8
28.4
32.1
28.2
44.2
49.3
44.0
53.3
MV-BART
400M
53.9
55.7
57.4
28.4
29.3
30.6
44.4
45.7
47.5
53.6
Coref-BART
400M
53.7
56.9
56.4
28.5
30.5
29.7
44.3
46.9
46.5
53.5
ConDigSum
400M
54.3
56.0
57.6
29.3
30.4
31.2
45.2
46.6
48.0
54.0
GPT-3-finetune
175B"
53.4
29.8
45.9
Alpaca
7B
28.2
26.0
39.8
5.7
5.1
8.3
20.5
19.2
29.0
19.4
Flan-T5-XXL.
11B
52.6
62.6
50.0
28.5
34.1
27.1
44.1
52.5
41.9
53.2
Flan-UL2
20B
53.3
60.3
52.5
28.0
32.0
27.7
44.1
50.0
43.3
53.5
ChatGPT
175B
32.7
22.4
70.2
12.3
8.4
27.1
24.7
16.9
53.6
32.5
InstructDS
3B*
55.3
58.8
57.5
31.3
33.5
32.6
46.7
49.7
48.6
55.5
w/ reference summary length
ChaiGPT
175B
40.8
39.3
43.4
13.7
13.2
14.6
31.5
30.5
33.4
40.0
InstructDS
3B*
58.4
58.5
58.8
32.8
32.9
33.0
48.9
49.0
49.2
58.5
Table 3: ROUGE scores on SAMSum test set. The results are divided into two blocks: dedicated dialogue summarization models and general- purpose LLMs. "w/ reference summary length" indicates reference summary lengths are provided in instructions. indicates 37.7M trainable parameters.
7.9k + 5k x 3) x 100k samples for training. It is important to note that all experimental results are obtained using the unified model without any specific tuning for individual datasets, which can potentially yield better results but at the cost of reduced generalizability. We employ LORA for parameter efficient training with total of 37.7 mil- lion trainable parameters (Hu et al., 2022).
et al., 2020), human evaluation and ChatGPT (GPT- 3.5-Turbo-0301) for comprehensive quality assess- ment.
Dialogue summarization can be approached using dedicated dialogue summarization models, which are specifically finetuned with in-domain data, well as general-purposss large language models (LLMs) that have larger model sizes and can perform various tasks by following instruc- tions. The selected benchmarking models are Dialogue Summarization Model. 1) Pointer- Generator (See et al., 2017), 2) BART (Lewis et al., 2020), 3) MV-BART (Chen and Yang, 2020), 4) Coref-BART (Liu et al., 2021 5) ConDig- Sum (Liu et al., 2021a), 6) GPT-3-finetuned (Hu al., 2022). General LLMss. 7) FLAN-T5 (Chung et al., 2022), 8) FLAN-UL2 (Tay al., 2022), 9) ALPPACA, and 10) ChatGPT.
4 Experiments
4.1 Datasets, Metries, and Baselines
We evaluate and benchmark our method on three dialogue summarization datasets including SAM- Sum (Gliwa et al., 2019), DialogSum (Chen et al., 2021) and TODSum (Zhao et al., 2021). These datasets equipped with dialogues and human-written verified summaries. Addition- ally, we explore dialogue reading comprehension with DREAM dataset (Sun et al., 2019) and we evaluate model accuracy without candidate choice exposures. In other words, we reframe DREAM an open question-answerti lem. The generated output is subsequently com- bined with BERTScore to determine the most pos- sible choice. More details and examples ex- plained in Section A.3. For evaluation metrics, we include ROUGE! (Lin, 2004), Score (Zhang
4.2 Main Results
The performance different models on the SAM- Sum dataset is presented in Table 3, providing insights into their capabilities for general dia- logue summarization. Notably, InstructDS outper- forms others and establish the new SOTA for both ROUGE and BERTScore metrics. In general, ded- icated summarization models show better perfor- mance because of their optimization specifically for the single task and dataset. In the case of gen- eral LLMs, Alpaca demonstrates less promising
generation prob-
WW utilize the ROUGF-score package from Google and compared with other implementations in Section A.I.

---

DualogSum
TODSum
Models
R-1
R-2
R-L
BS
R-1
R-2
R-L
BS
Alpaca
25.5
4.9
18.8
18.0
33.6
6.9
21.8
14.6
Flan- Flan-T5-Large TarLe
38.8
14.4
30.9
38.7
37.3
13.4
25.3
23.6
Flan-T5-XXL.
39.3
15.8
32.4
39.5
39.3
14.2
27.2
23.5
Flan-UL2
40.8
16.5
33.3
40.9
41.6
14.6
27.9
24.3
ChatGPT
38.4
12.9
29.8
38.8
39.8
11.8
24.5
24.9
BART
47.3
21.3
38.6
45.8
73.1
56.8
64.0
64.3
InstructDS
47.8
22.2
39.4
47.0
89.3
78.9
85.4
85.5
Table 4: Results on DialogSum and TODSum dataset. BART results are computed from the outputs released by Chen et al. (2021) and Zhao al. (2021).
Models
Multi-Choice Acc.
and underperform ChatGPT. is important to note that InstructDS is only directly trained with syn- thesized QDS triples from other datasets, whereas FLAN-based models are directly trained with the DREAM data. This demonstrates the effective- and generalizability of our synthesized triples. Further, we explored the impact of incorporating in-domain DREAM training data into InstructDS. This resulted a significant performance boost, surpassing ChatGPT by a considerable margin. To access the generalizability of InstructDS, we present results on DialogSum and TODSum datasets in Table 4. InstructDS maintenances its outstanding performance over other models. A significant gap exists between fine-tuned BART model and general-purpose I.LMs. In contrast, In- structDS incorporates multiple data sources and augmented QDS triples. This comprehensive dia- logue understanding framework contributes to its superior reasoning abilities across diverse dialogue domains and tasks.
Random
33.3%
Alpaca
51.3%
Flan-T5 Large
53.1%
Flan-T5-XXC
58.5%
Flan-UL2
56.8%
ChatGPT
60.8%
InstructDS
57.8%
+ +In-domain
65.9%
Table 5: Results on DREAM dataset. InstructDS is not trained with DREAM data. "+In-domain" indicates the inclusion of DREAM training data.
performance due to its optimization using synthe- sized instruction data, with limited involvement of dialogue summarization tasks. In contrast, as de- picted in Table 1, FLAN-based models include the SAMSum dataset in their instruction tuning pro- cess, resulting in competitive performance. While ChatGPT renowned for versatility across var- ious tasks, it prone to generate lengthy sum- maries when not constrained by prompts (Qin et al., 2023). Therefore, we further experiment with adding the reference summary length to instruc- tions during summary generation. This approach significantly improves the performance of Chat- GPT, achieving a balance between precision and recall. Meantime, InstructDS exhibits further per- formance boost, demonstrating its ability to follow length instructions.
4.3 Ablation Study
To provide insights into the effectiveness of In- structDS. we conduct ablation studies on In- structDS variants answer two fundamental ques- tions: 1) How augmented QDS triples contribute to general and query-based dialogue summariza- tion? 2) What is the effect of length awareness augmentation on general and length-controllabe dialogue summarization?
Model variants. We examine five model vari- ants of InstructDS across four evaluation datasets. Each variant trained with distinct training data: 1) MixDS: Mixing three dialogue summarization datasets, 2) MixDS+QDS: MixDS and synthe- sized QDS triples, 3) MixDS+Length: MixDS and
We conduct experiments on query-based dia- logue summarization using the DREAM dataset and the results are presented in Table 5. InstructDS can achieve comparable performance with Flan-T5

---

MIXDS
MiXDS+QDS
Lenghh
InstructDS
EI InstructDS+DREAM
70
86
65
35.3
64
60
36
$7.5
56.4
55
50
SAMSum
DREAM
DialogSum
TODSum
Figure 3: Ablation study different variants of InstructDS. Averaged ROUGE-1/2/L score is reported for dialogue summarization datasets, including SAMSum, DialogSum and TODSum. Accuracy is computed for DREAM dataset.
SAMSum ROUGE
Accusey
55
Awy,
SAMSum
with ref.
2500
5000
7500
10000
0.0
0.2
0.4
0.6
0.6
3.0
Triples
Length Augmentation
Figure Ablation study on the number of included QDS triples. Performance on SAMSum (left, averaged ROUGE) and DREAM (right, accuracy) datasets are reported.
Figure 5: Ablation study on the percentage of length augmented instances. Performance on standard SAM- Sum and length-revealed SAMSum are reported.
because TODSum summaries are more templated, requiring less reasoning. In summary, our findings demonstrate that augmented QDS triples enhance dialogue understanding and reasoning capabilities, enabling guery-based dialogue summarization with transferability to other dialogue datasets (such DREAM). Length augmentation, the other hand, enables only length controllability but also im- proves length awareness in general summarization tasks.
length augmentation, 4) InstructDS: MixDS, syn- thesized QDS triples and length augmentation, 5) InstruetDSDDREAM MixDS, synthesized QDS triples, human-written QDS triples from DREAM, and length augmentation. All other hyperparame- ters and evaluation protocols are kept identical for fair comparisons.
The ablation results are presented in Table 3. First, on SAMSum and DialogSum datasets, the full InstructDS model demonstrates the best perfor- mance. Incorporating both synthesized QDS triples and length augmentation techniques contributes an overall performance improvement. We at- tribute these performance boosts on the enhanced dialogue understanding and length awareness ca- pabilities. On DREAM dataset, the inclusion of synthesized QDS triples leads to an improvement in pused dialogue summarization performance, elevating it from 56.4 to 59.1. Notably, the per- formance is further enhanced when additional in- domain training data from the DREAM dataset. In TODSum dataset, we observe that augmented QDS triples do not yield better summarization results. However, the utilization of length augmentation techniques does improve the performance. It is
Number of QDA Triples. To examine the im- pact of varying numbers of augmented QDS triples on InstructDS, we conduct experiments on SAM- Sum and DREAM datasets, as shown in Figure 4. Consistent with previous observations, the inclu- sion of augmented QDS triples guides the model to reason more effectively, resulting in improved performance on general dialogue summarization. Additionally, the results from the DREAM dataset indicate that increasing the number of synthesized QDS triples leads to better performance in query- based summarization. As the number of QDS triples increases, the accuracy improves from 54 to over 58.
Length Augmentation Percentage. The impact of varying the percentage of added length augmen-

---

Human Annotator Informativeness
ChatGPT
Models
Faithfulness
Fluency
Conciseness
Faithfulness
Fluency
Informativeness
Conciseness
BART
3.85
4.36
3.22g.0
0.30
4.22
4.80
3,37
4,93,o.)
Alpaca
3.24
3.77
3.45
3.11(14)
3.59
4.07
3.19
4.29(..))
Flan-UL2
4.00
4.38,9)
3.03u.
4.29(1.)
4,45,09
4.78p.5)
(,52
4.91a33
ChatGPT
4.52mmm
4.38pm
4.62mm.
2.77
4,94
4.94
4.78,0.4
4.89az)
wumtnn-ritttnn
4.34
4.54,m
3,58n.
4.36,0.91
4.49/a8)
4,84
3,74,1.0:
4,95,a.3)
bustuceDDS
4.13
4.35ga
3.54
4.23
4.60
4.82
3.78
4.92(a.)
Table 6: Subjective quality evaluation with instances random sampled from SAMSum dataset (30 samples for Human Annotators, 200 samples for ChatGPT.). The Mean and standard deviation of evaluation scores reported.
tation samples is shown Figure 5. reveals that increasing the number of length augmentations can enhance the model's ability to control the gener- ated summary length while saturation point exists. Meantime, the effect on general summarization is diverse and relatively less consistent.
tions of the SAMSum dataset (Wang et al., 2022b; Gao et al., 2023b). ChatGPT's ability to generate detailed content, similar to the concept of Chain- Of- Thought (Wei et al., 2022), also contributes to higher faithfulness. Overall, InstructDS achieves comparable performance to human-written sum- maries in terms of fluency, informativeness, and conciseness. While InstructISS still falls short on human- faithfulness, it demonstrates notice- able improvements compared to previous models.
4.4 Subjective Quality Evaluation
We conduct multidimensional evaluations to ac- cess the quality of generated summaries. This in- volves fine-grained Likert scores (scale I to 5, the higher the better) from both human and ChatGPT in four dimensions: Faithfulness, Fluency, Infor- mativeness, and Conciseness (Wang al., 2022b; Gao et al., 2023a). Evaluations are performed on SAMSum dataset and we randomly sampled 30 instances for human evaluation and 200 instances for ChatGPT evaluation. The user interface and prompt template can be found in Figure 6 and Table 7, respectively. Each dialogue was accom- panied by one Human--ritten summary and five ones. We engaged 12 volun- teers, resulting in 792 labeled samples. On average. each summaayy pair receives assessments from 4.4 annotators. The mean and standard devia- tion Likert scores are presented in Table 6.
When using ChatGPT as an off- off-the-shelf evalua- tor, observe that InstructDS is achieving on on-pair or better performance over human written sum- maries on four dimensions. Especially for faithful- ness, InstructDS is outperforming els except for ChatGPT. However, is worth not- ing that ChatGPT exhibits biases towards its own outputs, resulting in potentially inflated evaluation scores. Similar patterns have also been found in other studies that involve ChatGPT evaluation. For example, Zhang et al. (2023) shows that ChatGPT always assigns higher scores to its own outputs when solving math problems, leading to signifi- cant concerns when using ChatGPT as an evaluator. Furthermore, in this work, we found that ChatGPT is not effective in evaluating the conciseness of summaries, which introduces a noticeable discrep- ancy compared to human evaluators in this aspect. We think it is because ChatGPT is not aware of the desired conciseness of summaries, which also attributes the phenomenon that it generating lengthy summaries. Further explorations are nec- essary for a robust ChatGPT evaluator in dialogue and other domains (Wang et al., 2023b).
other mod-
With human annotations, fluency is the best- performing metric. All models, except for Al- paca, demonstrate the ability to generate fluent sum- maries. Alpaca's 's relatively poor performance can be attributed to its unsupervised training and lim- ited exposure to dialogue data. For informativeness and conciseness, ChatGPT and Alpaca produce the most informative summaries but receive the lowest conciseness scores. These models tend to gener- ate longer summaries, including elaborate details, indicating limited understanding of the desired compressiveness. Faithfulness evaluation emerges as crucial factor in practical applications, where ChatGPT surpasses human performance. This can be attributed to potential inaccuracies the annota-
5 Discussions
5.1
Relationship with Query-focused Summarization
Query-focused summarization is closely related to our instructive dialogue summarization con-

---

cept (Vig et al., 2022). There are some similari- ties and differences. An ideal instructive dialogue summarization model should be capable of han- dling a wide range of instructions when generating summaries. As illustrated Figure 2, our current model can accommodate general dialogue sum- marization, query-based dialogue summarization,, and dialogue summarization with length control. We anticipate that the range of instructions will be expanded in future research, encompassing di- verse sets of instructions and multi- multi-round dialogue summarization scenarios. In the meantime, as a domain-specific model, we anticipate that instruc- tive dialogue summarization could exhibit emer- gent capabilities as shown in general instruction- tuned LLMs.
state-of-the-art performance across all four bench- mark datasets.
Limitations
Dialogue summarization is a label-intensive task that demands substantial supervision and the col- lection of human-written summaries, which is both challenging and inoource-intensvee. Moreover, the transferability of annotations across different dia- logue domains introduces additional complexity. Therefore, develop highly adaptable dialogue summarization model, leveraging unsupervised di- alogue data becomes crucial. However, is worth noting that InstructDS does not incorporate unla- belled dialogue data, leaving room for potential improvement.
Another important aspect to consider in dialogue data is privacy. The sensitive nature of dialogues can hinder the accessibility and public availabil- ity of diverse dialogue datasets. Therefore, future enhancements InstructDS should address pri- vacy concerns and explore the utilization ad- vanced learning techniques such as federated learn- ing, which can enable collaborative and privacy- preserving training processes.
5.2 Long Dialogue (Meeting) Summarization
Summarizing dialogues in meetings, especially long ones, is a challenging task that requires a model capable of processing extended sequences. One promising avenue for research involves ex- panding our current method for summarizing lengthy dialogues, which can improve query-based meeting summarization and comprehension. In- stead of relying the entire lengthy dialogue, approach generates queries from reference sum- maries. This approach addresses the difficulties using pre-trained language models for long dia- logue inputs in meeting scenarios. One of the obsta- cles in meeting summarization is the limited data availability, which limits the model's ability to gen- eralize across different domains. Nevertheless, our method has the potential to alleviate data scarcity issues in the context of summarizing lengthy meet- ings. Another challenge is that meeting summa- rization often associated with transcripts with ASR errors (Jiang et al., 2023), which effect is under-explored in existing research.
Automatically evaluating the quality of dialogue summarization poses significant challenges. Ac- quiring human annotators for model development is expensive and inefficient. Existing evaluation metrics heavily rely on ROUGE, and ChatGPT has emerged as a newly proposed method for evalu- ation. As discussed in Section 4.4, it still lacks transparency and robustness. Therefore, there is pressing need for more effective evaluation meth- ods specifically tailored for dialogue summariza- tion. Multilingual and multicultural evaluation is crucial since dialogues are frequently intertwined with local norms, slang, code-switches and cultural nuances (Wang et al., 2023a).
6 Conclusion
A Acknowledgement
this work, we present InstructDS, which is the first instructive dialogue summarization model that excels in both general dialogue summarization and query-based dialogue summarization. InstructDS generate high-quality query-based summariza- tion tailored to user requirements. This achieve- ment is made possible through a combination of datti-d training, synthesized QDS triples and length-awareness awareness augmentations. Experimental re- sults demonstrate that InstructDS establishes new
This research was supported by funding from the Institute for Infocomm Research (I²R), A *STARR Singapore. The computational work for this article was partially performed on resources of the Na- tional Supercomputing Centre (NSCC), Singapore.
References
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind

---

Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901.
Junxian He, Wojeiech Kryscinski. Bryan McCann. Nazneen Rajani, and Caiming Xiong. 2022. CTRL- sum: Towards generic controllable text summariza- tion. In Proceedings of the 2022 Conference on Em- pirical Methods in Natural Language Processing, pages 5879-5915, Abu Dhabi, United Arab Emirates. Association for Computational Linguisties.
Jiaao Chen and Diyi Yang. 2020. Multi-view sequence- to-sequence models with conversational structure for abstractive dialogue summarization. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language P Processing (EMNLP), pages 4106- 4118, Online. Association for Computational Lin- guistics.
Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang. Lu Wang. and Weizhu Chen. 2022. LoRA: Low-rank adaptation of large language models. In International Conference Learning Representations.
Yulong Chen, Yang Liu, Liang Chen, and Yue Zhang. 2021. DialogSum: A real-life scenario dialogue sum- marization dataset. In Findings of the Association for Computational Linguistics: ACI-IJCNLP 2021, pages
Lifu Huang. Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2019. Cosmos QA: Machine reading comprehension with contextual commonsense rea- soning. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natu- ral Language Processing (EMNLP-IJCNI) pages 2391 1-211,,,,, Hong Kong, China, Association for Com- putational Linguistics.
Online. Association for Computa- tional Linguistics.
Hyung Won Chung, Le Hou, Shayne Longpre, Bar- ret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang. Mostafa Dehghani, Siddhartha Brahma, et al. 2022. Scaling instruction-finetune language models. arXiv preprint arXiv:2210.11 11416.
Ridong Jiang, Wei Shi, Bin Wang. Chen Zhang, Yan Zhang, Chunlei Pan, Jung Jae Kim, and Haizhou Li. 2023. Speech-aware multi-domain dialogue state generation with ASR error correction modules. In Proceedings of The Eleventh Dialog System Tech- nology Challenge, pages 105-112, Prague, Czech Republic. Association for Computational Linguistics.
Hoa Trang Dang. 2006. Duc 2005: Evaluation of question- focused summarization systems. In Pro- ceedings of the Workshop on Cask-Focused Summa- rization and Question Answering, pages 48-55.
Pradeep Dasigi, Nelson F. Liu, Ana Marasović, Noah A. Smith, and Matt Gardner. 2019, Quoref: A read- ing comprehension dataset with questions requir- ing coreferential reasoning. In Proceedings of the 2019 Conference on Empirical Methods in Natu- ral Language Processing and the 9th International Juint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5925-5932, Hong Kong, China. Association for Computational Linguistics.
Grant H Kester. 2004. Conversation pieces: and communication in modern art. Univ of Califor- nia Press.
Community
Mike Lewis. Yinhan Liu, Naman Goyal, Marjan Ghazvininejad. Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequene pre-training for natural language generation, translation, and com- prehension. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, pages 7871-7880, Online. Association for Computa- tional Linguisties.
Mingqi Gao, Jie Ruan, Renliang Sun, Xunjian Yin. Ship- ing Yang, and Xiaojun Wan. 2023a. Human-like sum- marization evaluation with chatgpt. arXiv preprint 3xXiv:2004.0254
Chin-Yew Lin. 2004. ROUGE: A package for auto- matic evaluation of summaries. In Text Sunmariza- tion Branches Out, pages 74-81, Barcelona, Spain. Association for Computational Linguistics.
Mingqi Gao. Xiaojun Wan, Jia Su, Zhefeng Wang, and Baoxing Huai 2023b. Reference matters: Bench- marking factual eтror correction for dialogue sum- marization with fine -ra-graided evaluation framework. arXiv preprint
19.
Junpeng Liu, Yanyan Zou, Hainan Zhang, Hongshen Chen, Zhuoye Ding. Caixia Yuan, and Xiaojie Wang. 2021a. Topic-aware contrastive learning for abstrac- tive dialogue summarization. In Findings of the Asso- ciation for Computational Linguistics: EMNLP 2021, pages 1229-1222222 1243, Punta Cana, Dominican Republic. Association for Computational Linguisties.
Bogdan Gliwa, wona Mochol, Maciej Biesek, and Alek- sander Wawer: 2019. SAMSum corpus: human- annotated dialogue dataset for abstractive summa- rization. In Proceedings of the 2nd Workshop on New Frontiers in Sunmarization, pages 70-79, Hong Kong, China. Association for Computational Linguis- tics.
Zhengyuan Liu and Nancy Chen. 2021. Controllable neural dialogue summarization with personal named entity planning. In Proceedings of the 2021 Con- ference on Empirical Methods in Natural Language Processing, pages 92-106, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Chih-Wer Wee-Wen Goo and Yun-Nung Chen. 2018. Abstractive dialogue summarization with sentence-gate model- ing optimized by dialogue acts. In 2018 IEEE Spoken Language Technology Workshop (SLT), pages 735.. 742. IEEE.

---

Zhengyuan Liu and Naney Chen. 2022. Entity-based de- noising modeling for controllable dialogue summa- rization. In Proceedings of the 23rd Annual Meeting of the Special Interest Group on Discourse and Dia- logue, pages 407-418, Edinburgh, UK. Association for Computational Linguistics.
Xiangru Tang, Arjun Nair, Borui Wang, Bingyao Wang. Jai Desai, Aaron Wade, Haoran Li, Asli Celikyil- maz, Yashar Mehdad, and Dragomir Radev. 2022. CONFIT: Toward faithful dialogue summarization with linguistically-inte -ifformme contrastive fine-tuning. In P Proceedings of the 2022 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies. pages 5657-5668, Seattle, United States. Association for Computational Linguistics.
Zhengyuan Liu, Ke Shi, and Nancy Chen. 2021b. foreference-awarr dialogue summarization. In Pro- ceedings of the 22nd Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 509-519 (099-19,,00 Singapore and Online. Association for Computational Linguistics.
Yi Tay, Mostafa Dehghani, Vinh Q Tran, Xavier Gar- cia, Jason Wei, Xuezhi Wang. Hyung Won Chung, Dara Bahri, Tal Schuster, Steven Zheng. et al. 2022. U12: Unifying language learning paradigms. In The Eleventh International Conference on Learning Rep- resentations.
Preksha Nema, Mitesh M. Khapra, Anirban Laha, and Balaraman Ravindran. 2017. Diversity driven atten- tion model for query-based abstractive summariza- tion. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers), pages 1063-1072, Vancouver, Canada, Association for Computational Linguistics.
Jesse Vig, Alexander Fabbri, Wojeiech Kryscinski, Chien-Sheng Wu, and Wenhao Liu. 2022. Exploring neural models for query-focused summarization. In Findings of the Association for Computational Lin- guistics: NAACL 2022, pages 1455-1463 Seattle, United States. Association for Computational Lin- guistics.
Chengwei Qin, Aston Zhang, Zhuosheng Zhang, Jiaao Chen, Michihiro Yasunaga, and Diyi Yang. 2023. Is chatgpt general-purpose natural language process- ing task solver? arXiv preprint arXiv:2302.06477
Bin Wang, Zhengyuan Liu, Xin Huang, Fangkai Jiao, Yang Ding, Ai Ti Aw, and Nancy F Chen. 2023a. Seaeval for multilingual foundation models: From cross-lingual alignment to cultural reasoning. arXiv preprint arXiv:2309.0477 047666
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang. Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with unified text-to-text trans- former. The Journal of Machine Learning Research. 21(1):5485-55 55511.
Bin Wang, Chen Zhang, Chengwei Wei, and Haizhou Li. 2022a. A focused study on sequence length for dialogue summarization. arXiv preprint 2rXiv:2209.11911
Abigail See, Peter J. Liu, and Christopher D. Manning. 2017. Get to the point: Summarization with pointer- generator networks. In Proceedings of the 55th An- nual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1073- 1083, Vancouver, Canada. Association for Computa- tional Linguisties.
Bin Wang, Chen Zhang, Yan Zhang, Yiming Chen, and Haizhou Li. 2022b. Analyzing and evaluating faith- fulness in dialogue summarization. In Proceedings of the 2022 Conference Empirical Methods in Nat- ural Language Processing, pages 4897-4908, Abu Dhabi, United Arab Emirates. Association for Com- putational Linguistics.
Guokan Shang, Wensi Ding, Zekun Zhang, Antoine Tix- ier, Polykarpos Meladianos, Michalis Vazirgiannis, and Jean-Pierre Lorré 2018. Unsupervised abstrac- tive meeting summarization with multi sentti-eentenee com- pression and budgeted submodular maximization. In Proceedings of the 56th Anmual Meeting of the As- sociation for Computational Linguistics ( (Volue 1: Long Papers). pages 664 664, Melbourne, Australia. Association for Computational Linguistics.
Peiyi Wang. Lei Li, Liang Chen, Dawei Zhu, Binghuai Lin, Yunbo Cao, Qi Liu, Tianyu Liu, and Zhifang Sui. 2023b. Large language models are not fair evaluators. arXiv prepriat arXiv:2305. 17926.
Yizhong Wang. Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2022c. Self-instruct: Aligning language model with self generated instructions.
Dan Su, Tiezheng Yu, and Pascale Fung. 2021. Im- prove query focused abstractive summarization by incorporating answer relevance. In Findings of the Association for Computational Linguistics: ACL- IJCNLP 2021, pages 3124-3131, Online. Association for Computational Linguistics.
Yizhong Wang, Swaroop Mishra, Pegah Alipoormo- labashi, Yeganeh Kordi, Amirreza Mirzaei, Atharva Naik, Arjun Ashok, Arut Selvan Dhanasekaran. Anjana Arunkumar, David Stap. Eshaan Pathak, Giannis Karamanolakis, Haizhi Lai, Ishan Puro- hit, Ishani Mondal, Jacob Anderson, Kirby Kuznia, Krima Doshi, Kuntal Kumar Pal, Maitreya Patel, Mehrad Moradshahi, Mihir Parmar, Mirali Purohit, Neeraj Varshney, Phani Rohitha Kaza, Pulkit Verma. Ravsehaj Singh Puri, Rushang Karia, Savan Doshi,
Kai Sun, Dian Yu. Jianshu Chen, Dong Yu, Yejin Choi, and Claire Cardie. 2019. DREAM: A challenge data set and models for dialogue-based reading compre- hension. Transactions of the Association for Compu- lational Linguistics, 7:217-231.

---

Shailaja Keyur Sampat, Siddhartha Mishra, Sujan Reddy A, Sumanta Patro, Tanay Dixit, and Xudong General-
A Datasets and Metrics
Shen. 2022d. Super-Naturallnstretttn ization via declarative instructions on 1600+ NLP tasks. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. pages 5085-5109, Abu Dhabi, United Arah Emirates. Association for Computational Linguistics.
In this section, we will provide detailed expla- nation of the datasets and metrics that have been utilized in our work.
A.1 ROUGE Implementation
In the context of dialogue summarization, we ob- served that different studies incorporate different versions of the ROUGE metric, resulting in var- ied results. Specifically, we identified three widely used implementations of ROUGE:
Chengwei Wei, fun-Cheng Wang, Bin Wang, and C- C. Jay Kuo. 2023. An overview language models: Recent developments and outlook. arXiv preprint arXiv:2303.05759
Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoe V Le. and Denny Zhou. 2022. Chain of thought prompt- ing elicits reasoning in large language models. In Advances in Neural Information Processing Systems.
ROUGE-1.555 It is released by the author of the ROUGE paper. However, this imple- mentation has certain limitations. It requires running the Perl script, which can be inconve- nient when integrating with other projects. Additionally, the last update for this imple- mentation dates back to 2005.
Sarah Zhang, Samuel Florin, Ariel N. Lee, Ea- mon Niknafs, Andrei Marginean, Annie Wang. Keith Tyser, Zad Chin, Yann Hicke, Nikhil Singh, Madeleine Udell, Yoon Kim, Tonio Buonassisi, Ar- mando Solar-Lezama, and Iddo Drori. 2023, Explor- ing the mit mathematics and eecs curriculum using large language models.
Py-rouge: A full Python implementation of ROUGE metric. It produces the same result as the Perl implementation. However, the pack- age is not maintained since 2018. Because several dialogue summarization papers use this package, we also report the results using this package in this appendix.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. BERTScore: Evaluating text generation with bert. In International Conference on Learning Representations.
Lulu Zhao, Fujia Zheng, Keqing He, Weihao Zeng, Yue- jie Lei, Huixing Jiang. Wei Wu, Weiran Xu, Jun Guo, and Fanyu Meng. 2021. TODSum: Task-oriented dialogue summarization with state tracking. arXiv preprint
ROUGE-score: Another Python implementa- tion of ROUGE. It is released and maintained by the Google research team. Recently, Hug- gingFace incorporate this package into their evaluation metrics package. Because of the influence of Google research and Hugging- Face, we expect the package will receive sig- nificant attention across various research do- mains. Consequently, in this paper, we utilize this implementation to calculate the ROUGE score. It is also actively updated, with the latest update in March 2023.
Ming Zhong, Da Yin, Tao Yu, Ahmad Zaidi, Mutethia Mutuma, Rahul Jha, Ahmed Hassan Awadallah, Asli Celikyilmaz, Yang Liu, Xipeng Qiu, and Dragomir Radev. 2021. QMSum: A new benchmark for query- based multi-domain meeting summarization. In Pro- ceedings of the 2021 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5905-5921, Online. Association for Computational Linguistics.
Ben Zhou, Daniel Khashabi, Qiang Ning, and Dan Roth. 2019. "going on vacation" takes longer than "go- ing for walk": A study of temporal commonsense understanding. In Proceedings of the 2019 Confer- ence on Empirical Methods in Natural Language Pro- cessing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLPP pages 3363-3369, Hong Kong, China. Association for Computational Linguistics.
This section presents the main results using the Py-rouge package. The results for SAMSum dataset are presented in Table 8 while the results for DialogSum and TODSum are in Table 9. These results align with the patterns and conclusions dis- cussed in Section 4.2.
A.2 DialogSum Preprocessing
Haichao Zhu, Li Dong, Furu Wei. Bing Qin, and Ting Liu. 2022. Transforming wikipedia into augmented data for muery-focused summarization. IEEE/ACM Transactions on Audio, Speech, and Language Pro- cessing, 30:2357-2367.
In the original DialogSum paper (Chen al., 2021). the authors used #Person 1# #Person2#, and so on to represent speakers because the original dialogue did not contain speaker information. However, this

---

approach leads inconsistencies some names were already present in the original dialogue. To address this issue, we performed additional prepro- cessing on the data to align its format with SAM- Sum. Specifically, we employed the prompt tem- plate shown in Table 7 to prompt the FLAN-75-XL model to predict the name of the person. We then applied rule-based filtering determine the appro- priateness of the predicted names. This filtering process involved considering factors such for- bidden names labeled by humans, the length of the predicted name, the presence of special symbols, and whether the predicted name has appeared in the original dialogue. If the name did not meet the criteria according to our method, we used FLAN-T5-XL again with the tem- plate from Table 7 to choose from pool of ten candidate names, which consisted of five randomly sampled male names and five randomly sampled female names. Simultaneously. the name was cor- respondingly updated in the reference summary. Examples of preprocessed dialogues can be found in Table 1l and Table 16. To facilitate fu- ture research and development, we will make the name-replaced version available for public access.
evaluation for annotators. The annotator is asked label the quality of summaries four dimensions.
Table 7: the templates used for several tasks in- cluding query generation, text-based filtering, ChatGPT evaluation and the preprocessing on DialogSum dataset.
Table 10, and 12: shows the synthesized QDS triples including both kept and removed ones. The reason for filtering is also demon- strated.
Table 13, 14, 15, 16 and 17: shows case stud- ies on the generated summaries and query- based summaries on all four datasets includ- ing SAMSum, DREAM, DialogSum, and TODSum.
identification
A.3 Evaluation on DREAM
DREAM (Sun al., 2019) dataset is introduced for dialogue reading comprehension and the evaluation is designed as multi-choice questions. However, in real-world applications, where information queries are performed on dialogues, it is unlikely to have several candidate answers included as input. Real- world scenarios in dialogue reading comprehension are better represented as unconstrained text genera- tion problems.
evaluation of the DREAM dataset is con- ducted in an unconstrained manner, without provid- ing candidate choices to the model. To assess accu- racy, we utilize the BERTScore package to measure the similarity between the generated output and the answer choices, selecting the highest-scoring op- tion the final answer. The evaluation process is illustrated in Figure 7 and Figure 8.
B Templates and Examples
We provide more detailed illustrations of the human evaluation interface, instruction templates, synthe- sized QDS triples and case studies on model out- puts.
Figure 6: the interface used for subjective

---

Task Query Generation (Sec. 3.2)
Prompt Template
Generate answerable and specific question based on the following context. Context: S(Summary)
Can we get an answer from the context, yes or no? Question: s(Question) Context: ${Summary)
Text-based Filtering (Sec. 3.2)
Is the question fully answerable from the context without any guessing, yes or no? Question: $(Question) Context: S(Summary) Evaluate the quality the abstractive summary from the dialogue. Please be extremely picky. Rate each summary four dimensions: Faithfuliness: whether the summary is correct according to dialogue, Fluency: Whether summary grammarly correct, Informativeness: Whether the summary contains all essential information, Conciseness: Whether the summary is very concise (not verbose). Output should follow the template: 'Faithfulness': value, 'Fluency': value "Informativeness" value, 'Conciseness': value. You should rate on scale from (worst) to 5 (best). Do not give detailed explanations. Diologue $(Dialogue). Summary: $(Summary)
ChatGPT Evaluation
(Sec. 4.4)
(1) Who is #Person1# in the following dialogue? $(Dialogue} (2) Select on proper name for #Person1# from $(candidate names) in the following dialogue? $(Dialogue)
DialogSum Preprocessing (Sec. A.2)
Table 7: Prompting templates QDS triple generation, text-hased query filtering, ChatGPT evaluation, DialogSum preprocessing.
ROUGE-1
ROUGE-2
ROUGE-L
Models
F1
Pre
Rec
F1
Pre
Rec
Fi
Pre
Rec
BART
53.2
59.3
53.1
28.6
32.3
28.4
50.3
54.9
49.9
MV-BART
54.0
55.8
57.5
28.5
29.5
30.7
50.6
51.5
53.2
BORe
53.9
57.1
56.6
28.6
30.7
29.8
50.4
52.5
52.3
SonDigSun
54.4
56.2
57.8
29.4
30.5
31.4
51.3
52.3
53.7
Alpaca
28.3
26.1
40.0
5.7
5.2
8.4
26.5
24.6
34.9
-argg
51.3
58.7
50.1
25.9
29.7
25.4
48.7
54.2
47.3
Flan-T5-XXI
52.8
62.8
50.2
28.6
34.2
27.3
50.2
57.8
47.7
Flan-UL2
53.4
60.4
52.6
28.1
32.2
27.8
50.2
55.5
49.1
ChatGPT
32.9
22.5
70.6
12.4
8.5
27.3
31.6
22.4
59.3
InstructDS
55.6
59.1
57.8
31.4
33.6
32.8
52.7
55.1
54.2
w/ reference summary length
ChatGPT
40.9
39.5
43.5
13.8
13.2
14.7
37.7
36.6
39.5
InstructDS
58.6
58.7
59.0
33.0
33.1
33.2
54.5
54.6
54.8
Table SAMSum results using Py-rouge package.

---

DialogSum
TODSum
Models
R-1
R-2
R-L
R-1
R-2
R-L
Alpaca
25.6
4.9
24.6
33.4
6.9
28.1
Flan-T5: Plargee-Large
38.7
14.3
37.2
37.2
13.3
31.6
Flan-T5-XXX
39.3
15.8
38.7
39.2
14.1
33.7
Flan-UL2
40.7
16.5
39.6
41.5
14.6
34.4
ChatGPT
38.4
12.8
36.3
39.6
11.7
30.8
BART
47.2
21.1
44.8
73.1
56.8
68.7
InstructDS
47.7
21.8
45.3
89.2
79.8
87.5
Table 9: DialogSum and TODSum results using Py-rouge package.
Dialogue 56
Paul: What color flowers should
Cindy: any just yellow
Paul: ok, pink?
Cindy: no maybe red
Paul; just tell
type
Cindy: ugh, masel",
Paul and Cindys conversation about flowers. Cindi get yellow flowers, Paul asks her what colors she prefers. When she says red, he suggests rosestype and she responds with frustration.
(worst)
4
(beat)
Faithfulness
O
O
O
Fluency
0
0
0
O
Informativenees
0
0
Figure illustration of the user interface human evaluation summarization qualities.

---

Dialogue:
Tam, at your shoes. dirty they You clean
Oh, mum, but cleaned them yesterday.
W: They are dirty now. You must clean them again.
M: do not want to clean them today, Even if clean them today, they will get dirty again tomorrow.
right, then.
something eat, W: You had your breakfast in the morning, Tom, and you had lunch at school.
Mum,
M: am hungry again.
W: Oh, hungry? But if give you something to eat today, you will be hungry again tomarrow.
Query:
Why did the woman say that she wouldn't give him anything to eat?
Candidate Choices:
A: Because his mother wants to correct his bad habit.
B: Because he lunch school.
hungry.
Seq2Seq Model
Final Answer:
Because mather wants to correct his habit.
Figure 7: Constrained DREAM evaluation as multi-choices question answering.
Tom, at your shoes. dirty You Dialogue:
Oh, mum, but cleaned them only yesterday.
W: They are dirty now. You must clean them again.
M: do not want to clean them today. Even if clean them today, they will get dirty again tomorrow.
right,
M: Mum,
something eat, please. W: You had your breakfast in the morning, Tom, and you had lunch at school.
M: am hungry again.
Oh, hungry? But give you something eat today, you hungry again tomorrow.
Query:
Why did the woman say that she wouldn't give him anything to eat?
(1)
Candidate Choices:
Because his mother correct his bad habit.
Seq2Seq Model
Because he had lunch at school.
(3) BERTScore
(2)
hungry.
Output:
Because she want to educate her son on his eating habit.
(4)
Final Answer:
Because his mother wants correct his bad habit.
Figure 8: Unconstrained DREAM evaluation as open-ended question answering.

---

Dialogue (SAMSum)
Emmaa: We are going beach would you like to join in?
Sharol: sure who else is coming?
Enma: its anna, emily, wendy, kate and you
Sharol: would wendy be ok with joining..
Emma: think she would be ... is there any problem between you guys?
Sharol: think she doesnt like me she always try to avoid me...
Emma: really? then think you should definetly join and sort things with her Sharol: hmm.. dont want be her friend forcefully...
Emma: know she doesnt dislike you there must some misunderstanding...
lets meet up and sort out..
my place 11am
Sharol: sure will there
General Summary: Sharol is going to go to the beach with Emma, anna, emily, wendy and kate. Sharol is afraid that wendy doesn't like her.
Dialogue-Guided Question Generation
Query: Sharol will be at Emma's place at 11am.
Query: What is Sharol going to do?
Query: What is the problem between Sharol and Kate?
Query: What is Emma going to do?
Query: What will they do next?
Kept QDS Triples
Query: Who does Sharol go the beach with?
Summary: Sharol going to go the beach with Emma, Anna, Emily, Wendy and Kate.
Query: What is Sharol afraid of?
Summary: wendy doesn't t like her
Filtered QDS Triples
Query: Why is Sharol afraid Wendy doesn't 't. like her?
Reason for rejection: text-based filtering, the answer may not be answerable.
Query: Who is Sharol going to the beach with?
Reason for rejection: semantic-based filtering, duplicate question.
Query: What is Sharol's relationship to Wendy?
Reason for rejection: text-based filtering, the answer may not be answerable.
Table 10: One example from SAMSum dataset on the synthesized
triples.

---

Dialogue (DialogSum)
Byron: Are you busy tomorrow night? I'm going over to the Workers Club for volleyball if you'd ' like to come along. Jacquenette: Isn't that far away? Byron: Not really. If you take the if volleyball doesn't interest you, they' got huge indoor swimming pool, weight room, and indoor tracks. It's great place to meet people. Would you like to go with me?
bus, you can get there in ten minutes. And
Jacquenette: Now that you mentioned it, would be nice get away from the computer center for change, and I really should get some more exercise. Working up sweat in ball game or the weight room would be nice. I've got so fat. Byron: Just look at me! Yeu're not the only one. In high school had a lot more time to sports activities. Now what little spare time you have must be used in doing exercise. At least the club is open until 12 pm.
Jacquenette: I guess it's worth a visit. Did you say you' re going tomorrow night? Byron: Yeah.
Jacquenette: OK, I'll come, too. How about meeting me in front the cinema eight, and we'll catch the bus there.
Byron: Sure, see you then.
General Summary: Lance invites Daffy to play volleyball together tomorrow night. Daffy realizes it's time for more exercise so Daffy agrees. Lance begins to talk about how much exercise Lance did in high school and how little time Lance can put into exercise now. They decide to meet in front of the cinema.
Kept QDS Triples
Query: What do Byron and Jacquenette decide to do?
Summary: They think they have been getting too fat and decide to do more sports.
Query: What does Byron invite Jacquenette to do tomorrow night?
Summary: go over to the Workers Club for volleyball
Filtered QDS Triples
Query: What sport do Byron and Jacquenette decide to do?
Reason for rejection: semantic-based filtering, duplicate question.
Query: What sport do Byron and Jacquenette decide to do more of?
Reason for rejection: text-based filtering, the answer may not be answerable.
Query: What does Byron invite Jacquenette to do tomorrow night?
Reason for rejection: semantic-based filtering, duplicate question.
Table 11: One example from DialogSum dataset the synthesized Saayy-Dialggu-Suuna triples.

---

Dialogue (TODSum)
User: Can you find me Chinese restaurant please?
Agent: Sure. There
chinese restaurants in town. you looking for certain price range or location?
User: Yes. would like it to in the expensive price range.
have found that fit that description. Is there particular area you would like? User: I would like to book table for 6 at 15:15 on Tuesday.
Agent:
Agent:
able get table at Hakka in the north. The reference number is 7KW8MVL3 and they'll '1 keep the table for 15 minutes. Does that suit you?
User: Yes, thank you. That's all now. Bye!
Agent: Thank you for contacting and enjoy your meal at the hakka. Have great day.
General Summary: the user wants to book the restaurant seat for 6 people at 14:15 on tuesday. expensive.
the specialty the restaurant is chinese food
Kept QDS Triples
Query: What the price range of the restaurant?
Summary: the user wants to book the seat for people at 14:15 on tuesday. the specialty of the restaurant is chinese food
is expensive.
Query: What type of food does the restaurant serve?
Summary: chinese
Query: What type of restaurant is the user interested in?
Summary: chinese
Filtered QDS Triples
Query: What type of food is the restaurant?
Reason for rejection: semantic-based filtering, duplicate question.
Query: What type restaurant is the user looking for?
Reason for rejection: semantic-based filtering, duplicate question.
Table 12: One example from TODSum dataset the synthesized Laaa--iiioou--uummaa triples.

---

Case #1 from SAMSum
Dialogue:
Marsha: Guys, we've planned the trip with John last night as we promised
Cynthia: great, thank you that
Marsha: but of course you have to agree on that
Mohammad: sure, but really trust you
Gavin: me too
Marsha: so as decided last time, will spend week just on the beach
sha: we all wanted some calm, nice place, right?
Gavin: yes!!!
Marsha: John found this little hotel in Nosy Be
Marsha: <file_photo>>
Marsha: and their website: <file_other>
Marsha: it's quite basic the website, but it may actually be good sign
Gavin: I Love it!
Cynthia: seems just perfect
Marsha: I know
Marsha: it's
expensive than hotels on the mainland
Marsha: but it seems to be the most beautiful place in Madagascar
Mohammad: so let's go there, it's still cheaper than Italy for example Marsha: exactly!
Summary Human Reference
Marsha and John planned the trip. They will spend week the beach with Cynthia, Mohammad Gavin. They will stay hotel Nosy Be. Summary BART
Marsha, Cynthia, Mohammad Gavin are going Madagascar. They will spend a week on beach. John found a hotel in Nosy Be.
Summary Alpaca
small hotel John had found inNosiv Be, Madgascar. It was more expensivethan hoteles on main land, but it was the best place according to the group. They all agreed that it would be worth the money.
The group agreed spend their vacation on a beach
Summary Flan-ULL2
Marsha, Cynthia, Mohammad and Gavin are going to Madagascar. They will spend a week on the beach in Nosy
Summary ChatGPT
Marsha informs the group that John had planned the trip, and they all agreed to spend a week on the beach. She shows them picture of hotel in Nosy and its website, which they all like despite it being more expensive than hotels on the mainland. Mohammad mentions that still cheaper than Italy and everyone agrees to there.
Summary InstructISS
Marsha, John, Cynthia, Mohammad and Gavin will spend a week on the beach Nosy Be. It's more expensive than hotels on the mainland, but it's still cheaper than Italy.
Table 13: The first study from SAMSum dataset and the generated summaries from different models.

---

Case #2 from
Dialogue:
Ken: Hi, you?
Ang: Just peachy! You?
Ken: I'm okay...
Ang: Just okay? What's wrong?
Ken: Just stressed; work stuff, fighting with Brad, too much going on nom's.
Ang: Hang in there, it will get better!
Ken: I know, but it's lot.
Ang: Can anything help?
Ken: You are! Listening to me vent! LOL!
Ang: Are you at least doing anything fun this weekend?
Ken: Show Saturday night, then seeing the grandkids Sunday at the
Ang: Sounds great! That will cheer you up!
Ken: Gotta run, work calls. Love you!
Ang: Love too! Have fantastic day!
Ken: You too!
Summary Нuman Reference
feels stressed because work and fighting with Brad. There also too much going on at mom's.
Ken is going to show on Saturday night. On Sunday Ken is seeing the grandkids at the zoo.
Summary BART
Ken is stressed, because he has to deal with work issues and family issues. He going to see show on Saturday night and see the grandkids on Sunday.
Summary Alpaca
<noinput>>
Summary Flan-UL2
there is going on s. Ang this weekend. They will go to zoo with the grandkids on Sunday and see the show on Saturday.
is stressed out. fighting
Summary ChatGPT
Ken discusses with Ang about feeling stressed due to work, family issues and too much going on. Ang offers support and asks if there's anything they do to help. Ken expresses gratitudi tude for listening to vent. They discuss plans for the weekend, including show on Saturday and seeing grandkids the zoo on Sunday. They end the conversation by exchanging love and well wishes.
Summary InstructDS
Ken is stressed because of work, fighting with Brad and too much going at mom s. He will see show on Saturday night
the grandkids Sunday at the zoo.
Table second case study from SAMSum dataset and generated summaries from different models.

---

Case #1 from DREAM
Dialogue:
What does the term "alternative energy source" mean?
When we think energy or fuel for our homes cars, we think petroleum, a fossil fuel processed from oil removed from the ground, of which there is a limited supply. But alternative fuels
many things. Wind,
and water all be used create fuel.
Is the threat of running out of petroleum real?
M: It has taken thousands of years to create the natural stores of petroleum we have available at time. The real controversy surrounding the amount petroleum we have is how much we need to keep reserve for future use. Most experts agree that by around 2025, the amount of petroleum and availability will begin seri seriously decline. This not say there will be petroleum at this point. But it'll become very difficult and therefore expensive to extract.
much faster rate than it being produced
will reach peak. Then production
W: Is that the most important reason to develop alternative fuel and energy sources? very clear reasons fuels left, we have to find other fuel sources eventually. So the sooner we start, the better off will be. The other big argument is that when long time, which leads to some long-term global warming and greenhouse effect.
There
so. One that whether
60
years of fassil
negative effects,
Queries:
What do we usually refer when talk about energy according man?
(2) What do most experts ts agree on according to the man?
(3) What does the man think should do now?
Summary Human Reference
(1) refined from oil extracted from underground.
(2) 0il production will begin to decline worldwide by 2025.
(3) Start developing alternative fuels.
Summary Alpaca
(1) When we refer energy, according the Man, usually refers to petrol, natural gas, and other fassils fuells. These fuell sources are limited and will eventually run out. Alternative fuelles being developed reduce the dependence these limited fuellers.
(2) Most experTS agree THAT by AROUND
(3) The man thinks we must develop alternate energy and fuel source soon as possible because it
will help reduce dependence fossils fuel and limit the negative environmental impacts caused by burning them.
Summary - Flan-UIL2
(1) petroleum
(2) the
of petroleum will reach peak 2025
(3) Find alternative energy sources
Summary ChatGPT
(1) When we talk about energy according to the
usually refer to petroleum, fassil fuel the ground, which there limi ted supply.
processed from oil removed
(2) Most experts agree that by around 2025, the amount of petroleum we use will reach peak and then production and availability will begin to seriously decline.
(3) There no specific man mentioned the given context suggests something to do now. The conversation is about alternative energy sources and the need to develop them for sustainable future.
Summary InstructDS
(1) petroleum, fossil fuel processed from oil removed from the ground, of which there is a
limited supply
that by around 2025, the amount of petroleum will reach peak
(3) develop alternative fuel and energy sources
Table 15: One case study from DREAM dataset and the generated query- based summaries from different models.

---

Case #1 from DialogSum
Dialogue:
Whitaker: Hello, so how are we feeling today?
Jill: Things are going well for doctor.
Whitaker: Am I correct in thinking that you here for your annual physical?
Jill: Yes, am applying for new health insurance,
need physical examination to qualify.
whitaker: Your basic physical exan will include lungs, heart, blood levels, and eyes, ears, and nose. Jill: I've been having a little trouble breathing. Would you look into that, please?
Whitaker: We can do allergy test, and later I can send you for asthma test. When you give blood test, what are you looking for? Whitaker: going to check your cholesterol, blood sugar, white blood cell count.
Jill: would appreciate
Jill: I am expecting the tests to go well. I have been taking good care of myself.
Summary- Human Reference
Jill is going through an annual physical examination to apply for new health insurance, and Jill asks Whitaker look into the breathing.
Summary BART
#Person2# (Jill) comes to #Personi# (Whitaker) 's to have physical examination for new health insurance and asks for allergy test and asthma test.
Summary Alpaca
Doctor and patient discuss annual check-up, including labs, insulin, asthmatic tests, chlolestrol, sugar and count, allergies, healthy lifestyle, physical mental health.
Summary Flan-U1.2
Jill seeing Whitaker her annual physical. She needs qualify for new health insurance. Jilll has been having trouble breathing. Whi takker will check her cholesterol, blood sugar, and whi te
blood cell count.
Summary ChalGPT
Jill visits Whitaker for her annual physical and requests for Asthma test due to breathing problems. Basic physical includes checking of lungs, heart, blood levels, eyes, ears, and nose. Blood test will include cholesterol, blood sugar, and white blood count.
Summary InstructDS
Jill comes to Whitaker for Jill's annual physical because Jill needs a physical examination to qualify for new health insurance. Whitaker will do an allergy test and send Jill for an asthma test.
Table 16: One case study from DialogSum dataset and the generated summaries from different models.

---

Case #1 from TODSum
Dialogue:
User: Hi, can you help me plan Cambridge trip? need train from London Kings Cross to Cambridge.
Agent: I can help with that. One leaves friday at 05:17, is that time okay for you?
User: Actually, need leave Tuesday ofter 10:30.
Agent:
a train that leaves Tuesday at 11:17 that will arrive at 12:08. Would you like make à reservation?
User: Yes, please make a reservation for 6 people. I'll need the reference number.
Agent: I was able to book
train. Your reference number is F13RS8CP. Is there anything else that
you need?
User: I'm also looking for a place to eat. would like an expensive south indian restaurant.
Agent: It looks like we don't have any of those. Would you like to try something else?
User: Hmm, are you sure? I want something south indian, expensive and in the centre of town.
Agent: I have curry garden for Indian in the centre of town, but south indian. that serves medi terranean?
User: What about
two options. I can recommend la mimosa. Would you User: You said you have two options. what is the other option available?
Agent:
reserve a table?
Agent: Other than the La Mimosa Restaurant, I also have the Shiraz Restaurant. Both are located in the center and are expensive, Which do you prefer?
User: Lets go with the Shiraz please
Agent: perfect, would you like
reservation for you?
User: No. I think
ok although would like the postcode. Agent: The postcode is cb2ldp. Is there anything else can help you with today?
User: No, that is all. Thank you so much.
Agent: Let me know if you need anything else. great help. Goodbye!
User: I don't need anything else. You
Agent: Thank you for using our system!
Summary- Human Reference
the user wants the agent to help purchase train tickets. this train leaves at 10:30 on tuesday, from london kings cross to canbridge. the user asks for the postcode of the restaurant. the restaurant is tuated in the centre and has expensive medi food.
Summary- BART
user wants the agent to help purchase 6 train tickets. this train leaves at 10:30 on friday, from london kings cross to canbridge. the user wants to book the restaurant seat for 6 people at 12:15 on tuesday. the restaurant is located in the centre and the food there is expensive.
Summary- Alpaca
The user needs to travel from Kings to Cambrige and is looking to depart on Friday at the latest. The agent able offer 5. 15am train which acceptable the They also need accommodation for night before
agent suggests the Curry Garden for Indian food Cambridge, which they find acceptable. Finally, the travel agent provides the necessary reference numbers for their reservations. Summary Plan-UL2
Agent helps the user book train ticket from London Kings Cross to Cambridge. The user also wants to find an expensive restaurant in the centre of town.
Summary- ChaiGPT
The user asks for assistance in planning a trip to Cambridge. The agent helps with booking a train and restaurant reservation with alternatives and provides reference nunbers and postcodes. The user expresses gratitude and ends the conversation.
Summary InstructDS
book train tickets. this train leaves 10:30 tuesday, from London kings the postcode
restaurant. the restaurant offers the expensive medi terranean dishes, which is located in the centre.
to canbridge. user
Table 17: One case study from TODSum dataset and the generated summaries from different models.


Reviews:
[{"review_id": "SxUwz8miEn", "reviewer": "Reviewer_YtPF", "summary": "", "questions": "Is there any difference between instructive dialogue summarization and query-focused summarization?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We want to express our appreciation for the thorough review of our paper and recognition of the satisfactory aspects of our work, such as its novelty, comprehensive analysis and performance improvements. We are also grateful for your insightful questions and we intend to address them point by point.\n\n* Can the proposed method enhance the ability of PLMs for understanding lengthy meeting dialogues?\n\n    * Certainly, our method can be directly applied to lengthy dialogue summarization datasets and improve the quality of query-based summarization and dialogue understanding. This capability arises from the fact that our proposed method is not constrained by dialogue length. To clarify, our approach involves the automatic generation of queries from reference summaries rather than replying on the lengthy dialogue itself. The filtering and query-based summary generation processes are all conditioned on the summary. This approach alleviates the challenges associated with using off-the-shelf PLMs to process lengthy dialogue inputs, which we have found to be suboptimal.\n\n    * It's worth noting that the focus of our paper is on daily dialogues, and we have conducted experiments using four popular datasets: SAMSum, DialogSum, TODSum, and DREAM. Regarding meeting summarization, as discussed in related work, we are aware of QMSum, which introduces a human-annotated query-based summarization dataset. However, meeting summarization datasets typically have limited samples due to the challenges of collection and transcription, which restricts their applicability to various meeting scenarios. Therefore, there is a pressing need for larger-scale meeting summarization datasets. Nevertheless, given the current circumstances, our proposed method can still alleviate the data scarcity issue.\n\n* Is there any distinction between instructive dialogue summarization and query-focused summarization?\n    * Yes and no. The term \"instructive dialogue summarization\" was introduced in our work. An ideal instructive dialogue summarization model should be capable of handling a wide range of instructions when generating summaries. As illustrated in Figure 2, our current model can accommodate general dialogue summarization, query-based dialogue summarization, and dialogue summarization with length control. We anticipate that the range of instructions will be expanded in future research, encompassing diverse sets of instructions and multi-round dialogue summarization scenarios. In the meantime, as a domain-specific model, we anticipate that instructive dialogue summarization could exhibit emergent capabilities as shown in general instruction-tuned LLMs.\n    * Concerning query-based summarization, most existing settings closely resemble dialogue question-answering. The summary is generated to address specific queries. However, this concept can also be enriched by considering various inquiries as queries. In this case, the concepts of instructive dialogue summarization and query-focused summarization become closely aligned. We will incorporate this discussion into the updated manuscript.\n\n* Missing references\n    * Thank you for pointing out the relevant papers. They will serve as supplementary references in addition to our current citations. We have already referred to some papers on controllable summaries and query-based summarization, and we will include the ones you've listed, along with necessary discussions.\n\nOnce again, we extend our gratitude for your efforts and for recognizing the contributions of our work. Your comments are helpful in further improving the quality of our paper. By the way, please be aware of a caching issue in the 'Paper topic' section.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The proposed instruction tuning method helps PLMs outperform larger language models in various dialog understanding and summarization datasets. \n\nThe experiments in this paper are comprehensive. It manually verifies the quality of a subset of synthetically generated instruction data. It utilizes both n-gram-based metrics, LLM-based metrics, and human evaluation to evaluate the qualities of generated summaries. It also conducted a detailed ablation study to evaluate the contribution of synthetic QDF triples and length-aware augmentation. \n\nThe proposed method for generating instructional dialogue summarization data is novel and interesting. It helps alleviate the scarcity of query-based dialogue summarization.", "reasons_to_reject": "It is not clear whether the proposed instruction tuning method can enhance the ability of PLMs to understand lengthy meeting dialogs. This paper should provide more discussion on this aspect. \n\nThe differences between instructive dialogue summarization and query-focused summarization are not clearly described. The authors should provide more discussions on the differences if they do not refer to the same concept.", "questions_to_authors": "Is there any difference between instructive dialogue summarization and query-focused summarization?", "paper_topic_and_main_contributions": "This paper proposes a novel instruction-tuning method to expand the capability of dialogue summarization models. It devises a three-step pipeline approach to generate a large-scale dialogue summarization training dataset with the instruction demonstration style. Experiment results show that the proposed instruction tuning method helps PLMs outperform larger language models in various dialog understanding and summarization datasets.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "This paper should include the following citations to recent papers on query-focused summarization:\n\nSumREN: Summarizing Reported Speech about Events in News. AAAI 2023\n\nExploring Neural Models for Query-Focused Summarization. NAACL 2022\n\nControllable Summarization with Constrained Markov Decision Process. TACL 2021"}}, {"review_id": "1SjSlvqy7O", "reviewer": "Reviewer_hwUr", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your thoughtful assessment of our paper. Your recognition of the paper's clarity, motivation, design of InstructDS and experimental analysis is encouraging. We are committed to addressing your concerns regarding the technical contribution and error analysis of our proposed approach in a comprehensive manner:\n\n* **Problem significance**: It is important to emphasize the importance of instructive dialogue summarization. Unlike conventional dialogue summarization models, our approach can adapt to users' interests with flexibility. Previous attempts in news and meeting summarization have introduced new human-annotated datasets to tackle this problem, but they suffer from resource-intensive and scalability challenges. In our paper, we introduce a novel method for instructive dialogue summarization by synthesizing high-quality QDS triples and training a unified model. The InstructDS model demonstrates superior performance across four widely used datasets and contributes to enhanced dialogue understanding.\n\n* **Insights**: We acknowledge that simply prompting Large Language Models (LLMs) for data synthesis may not be as interesting or applicable to our scenario. In the context of InstructDS, the task demands a deep understanding of task properties and model capabilities. Prompting an LLM for instructive dialogue summarization is not straightforward. First, we generate queries by mining information directly from carefully crafted human summaries, which involve complex reasoning processes. These generated queries are less likely to be superficial or extractive from the dialogues, thus guiding the model towards better reasoning capabilities. Furthermore, we employ two innovative filtering approaches, text-based and semantic-based, to ensure quality and diversity in the collected QDS triples. The quality of these triples is verified by human experts. Finally, we train a unified model by aggregating all datasets and synthesized components.\n\n* **Strong evidence and extensive analysis**: We have conducted thorough experiments on four widely recognized datasets: SAMSum, DialogSum, TODSum, and DREAM. Our results outperform all existing models in these datasets. Additionally, we present an ablation study to demonstrate the effectiveness of each proposed component. To assess subjective quality, we utilize both human evaluation and ChatGPT evaluation. For error analysis, the ablation study provides strong evidence, and we include examples of synthesized QDS triples and model predictions in the appendix.\n\n\nThanks again for your meticulous evaluation of our work. We would like to emphasize that instructive dialogue summarization is a non-trivial problem and our solution, which introduces synthesized triples based on a comprehensive understanding of various aspects, represents an effective approach to address this challenge.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is well-written and motivated.\nThe paper introduces InstructDS, the dialogue summarization model that excels in following user instructions explicitly. \nThe method harnesses the question generation and answering capabilities of Large Language Models (LLMs).\nExtensive evaluation of the proposed approach on three dialogue summarization datasets and one dialogue comprehension dataset shows that the results clearly illustrate a significant improvement compared to previous models.\nWell-performed ablation study and case studies proves the efficacy of the approach.", "reasons_to_reject": "The paper has limited technical contribution as the approach simply performs prompting LLMs.\nError analysis is missing.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The authors propose a comprehensive three-step approach for synthesizing high-quality query-based summarization triples. This process involves three key stages: summary-anchored query generation, query filtering, and query-based summary generation. To achieve this, they train a unified model named InstructDS (Instructive Dialogue Summarization) on three diverse summarization datasets containing multi-purpose instructive triples. To validate the effectiveness of the method, the authors conduct evaluations on four datasets, encompassing both dialogue summarization and dialogue reading comprehension tasks. Experimental results showcase the superiority of the approach over state-of-the-art models, including those with larger sizes. Moreover, the model exhibits higher generalizability and faithfulness, as corroborated by subjective evaluations conducted by human assessors.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "CwbTEiR1Hh", "reviewer": "Reviewer_M5kz", "summary": "", "questions": "The following are suggestions which I think can help strengthen the paper further. \n\n1. While I like the idea of conditioning on reference summaries, it could benefit the paper to formally investigate the quality of data generated when conditioning on the dialogues versus reference summaries. Currently, the paper makes the claim that it is challenging for models to generate data when conditioning on dialogues and empirical support would strengthen the paper.\n\n2. QMSum is one of the primary work on query-focused summarization, which is of great importance given the nature of long meeting conversations and hence the value added by doing query-focused summarization over generic summarization. It would be interesting to see if the synthetic data generation approach could help improve performance on such a task of meeting summarization as well. The datasets currently investigated in the paper (SAMSum, TODSum, DialogSum) contain much shorter conversations (in terms of number of utterances per speaker and total number of utterances) compared to conversations in QMSum, where the benefits of query-focused summarization are much more pronounced. Such investigation can further help strengthen the claims of the paper and shed light on benefits of LLM-based synthetic data generation+instruction-tuning. ", "limitations": "", "rating": 4, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your thoughtful and thorough assessment of our paper on instructive dialogue summarization. Your positive feedback and recognition of the significance of our work are motivating. We are particularly gratified that you found our approach to synthetic data generation innovative and the design of our experiments valuable. We will definitely take your suggestions and make adjustments to further strengthen the paper:\n\n* **Comparison between Dialogue-Guided Query Generation (DQG) and Summary-Guided Query Generation (SQG)**: We agree with your recommendation to provide a more thorough comparison between DQG and SQG in our updated manuscript. Specifically, we have observed that DQG tends to generate superficial questions and focus on individual or limited utterances. In contrast, SQG leverages the concise information contained within human-written summaries, requiring a higher level of human reasoning and commonsense. Furthermore, DQG's performance is hindered by the inherent limitations of large language models in processing lengthy content, leading to suboptimal results. SQG queries, on the other hand, are formulated to be answerable from the summary itself, reducing uncertainty. We will expound upon the rationale behind our design choices and include a qualitative study to substantiate our claims.\n\n* **Extending from Daily Dialogue Summarization to Lengthy Meeting Summarization**: Your suggestion to explore the applicability of our approach to lengthy meeting summarization is well-received. We believe that our methodology can seamlessly extend to summarizing lengthy dialogues, thereby enhancing query-based summarization and dialogue comprehension. Our approach generates queries from reference summaries rather than relying on the entire lengthy dialogue. This approach avoids the challenges of using pre-trained language models for lengthy dialogue inputs. The major hurdle for meeting summarization lies in the limited availability of samples, which hinders its generalizability to different domains. Nevertheless, our method can contribute to mitigating data scarcity issues in the context of lengthy meeting summarization. We will introduce a dedicated discussion on this topic and considering the space limitation of the current manuscript, a thorough exploration might have to be left as future research. Meantime, more challenges will be encountered for meeting summarization and new solutions should be adopted.\n\n\nThanks again for your thorough comments. Your insights will certainly guide us in further refining our work. We will improve the logic flow throughout the paper including the one regarding Line 067.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The study is thoroughly conducted with extensive number of summarization models, ablation studies and careful evaluations. \n2. Query-focused summarization has many practical applications and the author's efforts in this direction are much desired.\n3. I found it really interesting the way authors designed the synthetic data generation framework. For instance, conditioning on original reference summaries provided with existing datasets (instead of original dialouge) to curate the synthetic queries and focused-summaries is neat. As a result, the data generation process is slightly different from what the model is tasked to do at inference time, and hence the model is not generating data and making inferring for same task. As I understand, the model is generating questions and answers from a context (reference summary) which can then be recasted as a dataset for query-focused dialouge-summarization task. \n4. The authors provide detailed human and ChatGPT-based evaluations to support their conclusions. Overall, the paper is also well written and easy to read. ", "reasons_to_reject": "I don't think I have any major concerns. ", "questions_to_authors": "The following are suggestions which I think can help strengthen the paper further. \n\n1. While I like the idea of conditioning on reference summaries, it could benefit the paper to formally investigate the quality of data generated when conditioning on the dialogues versus reference summaries. Currently, the paper makes the claim that it is challenging for models to generate data when conditioning on dialogues and empirical support would strengthen the paper.\n\n2. QMSum is one of the primary work on query-focused summarization, which is of great importance given the nature of long meeting conversations and hence the value added by doing query-focused summarization over generic summarization. It would be interesting to see if the synthetic data generation approach could help improve performance on such a task of meeting summarization as well. The datasets currently investigated in the paper (SAMSum, TODSum, DialogSum) contain much shorter conversations (in terms of number of utterances per speaker and total number of utterances) compared to conversations in QMSum, where the benefits of query-focused summarization are much more pronounced. Such investigation can further help strengthen the claims of the paper and shed light on benefits of LLM-based synthetic data generation+instruction-tuning. ", "paper_topic_and_main_contributions": "The paper aims to improve query-focused dialouge summarization models by leveraging synthetically generated query-dialouge-summary triples via Flan-T5-XL and use it to instruction-tune the same model (Flan-T5-XL). The authors propose a method to generate such synthetic data based on question-generation (query) and answer-generation (focused-summary) from reference summaries for a given dialouge. The authors curate synthetic data from three existing dialouge summarization datasets which help improve generalizability of final instruction-tuned model. Finally, authors also experiment with summary-length constraints in the instructions, and observe improvements over unconstrained version. With experiments, authors demonstrate that synthetic-data based instruction-tuning helps improve performance. The authors also support their analysis with human and ChatGPT-based evaluations along various subjective dimensions (e.g., fluency, faithfulness, informativeness, conciseness).  ", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "Suggestions\n1. Line 067. The introduction provides motivation for research on query-focused summarization in paragraphs 2-3. However, it suddenly switches to the challenge of instruction-based dialouge summarization in paragraph 4 (line 607). The flow could benefit by bridging this gap with one line motivating use of instruction-based training (e.g., Recent work has shown promising results with instruction-based training for many task [CITE], and has the potential to benefit query-focused summarization. However, the challenge [...] is scarcity of data.)\n"}}]

Author rebuttal:

Meta-review: The paper introduces the idea of instructive dialogue summarisation, which combines dialogue QA, general summarisation and query-focused summarisation into a single task. All reviewers thought: (1) this idea is novel and well-motivated; (2) the paper is well-written and easy to understand; (3) the experiments are extensive and thorough, including multiple datasets, summarisation models, automatic and human evaluation; and (4) the approach of training data generation via LLM is intuitive, creative, and impactful. There are no major concerns raised, although the paper could benefit from the following minor revisions: (1) clarifying the task (instructive dialogue summarisation) better, in particular how it is related/different to query-focused summarisation; (2) discuss why QMSum is not included as one of its datasets (seeing it's very relevant to the task); and (3) discuss or provide preliminary results on the impact of using dialogues vs. references for generating training data.
Decision: Accept-Main