SuperDialseg: A Large-scale Dataset for Supervised Dialogue Segmentation
Authors: Junfeng Jiang
ChengZhang Dong
Sadao Kurohashi
Akiko Aizawa
Venue: EMNLP 2023 Main (2023)
Keywords: Dialogue Segmentation
Dataset
Benchmark
TL;DR: This paper provides a large-scale dataset, SuperDialseg, for supervised dialogue segmentation and an extensive benchmark for future research.

Abstract: Dialogue segmentation is a crucial task for dialogue systems allowing a better understanding of conversational texts. Despite recent progress in unsupervised dialogue segmentation methods, their performances are limited by the lack of explicit supervised signals for training. Furthermore, the precise definition of segmentation points in conversations still remains as a challenging problem, increasing the difficulty of collecting manual annotations. In this paper, we provide a feasible definition of dialogue segmentation points with the help of document-grounded dialogues and release a large-scale supervised dataset called SuperDialseg, containing 9,478 dialogues based on two prevalent document-grounded dialogue corpora, and also inherit their useful dialogue-related annotations. Moreover, we provide a benchmark including 18 models across five categories for the dialogue segmentation task with several proper evaluation metrics. Empirical studies show that supervised learning is extremely effective in in-domain datasets and models trained on SuperDialseg can achieve good generalization ability on out-of-domain data. Additionally, we also conducted human verification on the test set and the Kappa score confirmed the quality of our automatically constructed dataset. We believe our work is an important step forward in the field of dialogue segmentation.

SuperDialseg: A Large-scale Dataset for Supervised Dialogue Segmentation
Junfeng Jiang'
Chengzhang Dong
Sadao Kurohashi Akiko Aizawa
'The University of Tokyo KKyoto University NNational Institute of Informatics
jiangjfßis. seriis.s.ur siangjfeis.s.uutooke... w.u-tokyo.acc.o.
[dong@nlp.ist. is kuro@i} www.t-o.oa...... aizawa@nii.ac.
Abstract
discussed topics, and therefore benefits the dia- logue understanding for various dialogue-related tasks, including dialogue summarization (Liu et al., 2019b; Chen and Yang, 2020; Feng et al., 2021b; Zhong et al., 2022), response generation (Xie et al.. 2021), response selection (Xu et al., 2021), knowl- edge selection (Yang et al., 2022), and dialogue in- formation extraction (Boufaden al., 2001). When using Large Language Models (LLMs) dialogue systems (c.g.. ChatGPT)), dialogue segmentation methods can also help us to save tokens during long-term conversations while minimizing informa- tion loss. Figure shows an example of dialogue segmentation.
Dialogue segmentation is crucial task for di- alogue systems allowing a better understand- ing of conversational texts. Despite recent progress in unsupervised dialogue segmenta- tion methods, their performances are limited by the lack of explicit supervised signals for training. Furthermore, the precise definition segmentation points conversations still remains as a challenging problem, increasing the difficulty of collecting manual annotations. this paper, we provide feasible definition of dialogue segmentation points with the help document- document-grouunded dialogues and release large-scale supervised dataset called Su- perDialseg, containing 9,478 dialogues based on two prevalent document- groounddd dialogue corpora, and also inherit their useful dialogue- related annotations. Moreover, we provide benchmark including 18 models across five categories for the dialogue segmentation task with several proper evaluation metrics. Em- pirical studies show that supervised learning is extremely effective in in-domain datasets and models trained on SuperDialseg can achieve good generalization ability out-of-domain data. Additionally, we also conducted hu- man verification on the test set and the Kappa score confirmed the quality of our automati- cally constructed dataset. We believe our work is an important step forward in the field of dialogue segmentation. Our codes and data can be found from: https://gtt thub. com/ Coldog2333/SuperDiaassss
Owing to the scarcity of supervised dialogue segmentation datasets, previous studies developed unsupervised methods (Song et al., 2016; Xu al., 2021: Xing and Carenini, 2021) but easily reached the ceiling. Noting that models trained with large- scale supervised text segmentation datasets are ef- fective for segmenting documents (Koshorek et al., 2018; Arnold et al., 2019; Barrow et al., 2020), however, Xie et al. (2021) experimented that ap- plying them directly on dialogues were inadequate because dialogue has its own characteristics.
Previous studies showed that deep learning (DL) models usually perform better with supervised training signals and converge faster (Krizhevsky et al., 2017: Koshorek et al., 2018). Therefore, despite the expensive cost of manual annotation, previous works still attempted to collect some an- notated datasets. However, these datasets are either small, containing only 300 600 samples for train- ing (Xie al., 2021; Xia al., 2022), or not open- sourced, due the privacy concerns (Takanobu et al., 2018) and other issues (e.g., copyright issues from the industry (Xia et al., 2022)). Therefore, a large and publicly available supervised dataset is urgently needed. In addition, some studies on spoken dialogue systems provided manual annota-
1 Introduction
Recently, the importance of dialogue segmenta- tion for dialogue systems has been highlighted in academia (Song et al., 2016; Xing and Carenini, 2021; Gao et al., 2023) and industry (Xia et al., 2022). Dialogue segmentation aims to divide a dialogue into several segments according to the
"Corresponding Author

---

Agent
User
U1: Hello. like to leam about your retirement program, of
A2: you looking for family benefits?
U3: Not necessarily,
Medecin Faedly
teatt riter's wies
A4: Would you like learn about maximum family benefits?
Kzadio
US: Sure.
Dinetu for
A6: Do any of your children quality for benefits?
Bou's
U7: believe so.
A8: As far maximum family benefits, there upper payout limit.
U9: Will my child receive the full payment up front or monthly payments?
A10: your child qualifies, or receive monthly payments.
aroviie fat neeve sout
Figure I: A secument-ggrundee dialogue example selected from the doc2dial dataset (Feng et al., 2020). Different colors correspond different dialogue segments.
tions on speech data (Janin et al., 2003; McCowan et al., 2005), but because of the noisy transcripts of spoken conversations, these corpora also not suitable for training dialogue segmentation models.
In summary, our contribution is threefold.
We constructed a large-scale supervised di- alogue segmentation dataset, SuperDialseg, from two DGDS corpora, inheriting useful dialogue-relate annotations.
Apart from the cost, another challenge for con- structing a high-quality annotated dataset is the vague definition of segmentation points in dia- logues. In contrast to documents, which have clear structures such sections and paragraphs, seg- menting dialogues is particularly confusing and difficult. Existing annotated datasets generally suf- fer from disagreement among annotators (Xie et al., 2021), resulting in relatively low Kappa scores? or boundary ambiguity (Xia et al., 2022). Inspired by Xing and Carenini (2021), we conjecture that the document-grounded dialogue system (DGDS) (Ma et al., 2020) can provide a criterion for defining the boundary of dialogue segments. Specifically, in document-grounded dialogue, each utterance is grounded on piece of relevant knowledge within a document, illustrating what speakers concentrate on or think about when talking. This information can reflect the discussing topic for each utterance. Based on this characteristic, we construct a super- vised dialogue segmentation dataset, SuperDialseg. based on two DGDS datasets, containing 9,478 dialogues. Moreover, inherit their rich dialogue- related annotations for broader applications.
We conducted empirical studies to show the good generalization ability of SuperDialseg and emphasized the importance of considering paalogue-specifie characteristics for dialogue segmentation.
We provide an extensive benchmark including 18 models across five categories. We believe it is essential and will be valuable for the dia- logue research community.
2
Related Work
2.1
Text Segmentation
Before dialogue segmentation, text segmentation was already promising task for analyzing doc- ument structures and semantics by segmenting long document into several pieces or sections. To the best of our knowledge, cue phrases were used for text segmentation as early as 1983 (Brown and Yule., 1983; Grosz and Sidner, 1986; Hirschberg and Litman, 1993; Passonneau and Litman, 1997). Subsequently, Hearst (1997) proposed TextTiling to distinguish topic shifts based on the lexical co- occurrence. Choi (2000) and Glavaš et al. (2016) also proposed clustering-basee and graph-based methods for text segmentation. Before large-scale datasets existed, these unsupervised methods per- formed relatively well but casily reached the ceil- ing. Facing this challenge, Koshorek et al. (2018)
Usually, dialogue segmentation is considered as a subtask that aids other downstream tasks, and thus, it lacks standard and comprehensive bench- mark for comparison. Therefore, in addition to proposing the supervised dataset, conducted an extensive comparison including 18 models across five categories using appropriate evaluation metrics.
"TIAGE's Kappa: 0.479 (0.41-0.6: moderate agreement)

---

collected 727k documents from Wikipedia and seg- mented them based on the natural document struc- tures. With the Wiki727K dataset, they formu- lated text segmentation as a supervised learning problem and consequently led to many outstand- ing algorithms (Arnold et al., 2019; Barrow et al., 2020; Glavaš and Somasundaran, 2020; Gong et al.. 2022). However, previous work (Xie et al., 2021) experimented that it is inadvisable directly apply these studies to dialogue systems because dialogue possesses unique characteristics.
ble, but the dialogue flow around the segmentation point is not natural, resulting semantic fractures. Though the dataset can still serve as an evaluation dataset, this characteristic can be interpreted as shortcuts (Geirhos et al., 2020), leading to poor generalization in real-world scenarios. Based on prevalent DGDS dataset, doc2dial (Feng et al., 2020), Xing and Carenini (2021) used the train- ing and validation sets construct evaluation dataset for dialogue segmentation but missed some crucial details. For example, as shown in Figure 1, the user begins the conversation with a general query referring to the document title but the fol- lowing utterance from the agent refers to the first section. In this case, Xing and Carenini (2021) mis- takenly assigned them to different segments, even though they still discussing the same topic. Fur- thermore, most existing datasets neglected useful dialogue-related annotations like role information and dialogue act, while we inherit these annotations from the existing DGDS corpora to support future research. We will describe our dataset construction approach in Section 3.2. Table I summarizes the datasets mentioned above.
2.2 Document-grounded Dialogue System
Recently, limitations such as lacking common sense (Shang et al., 2015) and knowledge hallu- cination (Roller et al., 2021) have been identified in dialogue systems. To solve these issues, previous studies focused on extracting useful information from knowledge graphs (Jung al., 2020), im- ages (Das et al., 2017), and documents (Zhou et al., 2018; Dinan et al., 2019), yielding informative re- sponses. Feng et al. (2020, 2021a) proposed two DGDS corpora with rich annotations, including role information, dialogue act, and grounding ref- erence of each utterance, which enable knowledge identification learning and reveal how interlocutors think and seek information during the conversation.
Dataset
Reference
#Train Ex-Anno Open
SmartPhone Clothing
(Takanobu et al., 2018)
2.3 Dialogue Segmentation
DAct
(Zhang and Zhou, 2019)
20,000'
20,000
Unlike documents that have natural structures (e.g., titles, sections, and paragraphs), dialogue segmen- tation involves several challenges, including the vague definition of segmentation points, which in- creases the difficulty of data collection even for humans. Recently, Xia et al. (2022) manually collected 562 annotated samples to construct the CSTS dataset, and Xie et al. (2021) constructed the TIAGE dataset containing 300 samples for training. Not only the size of annotated datasets too small for supervised learning, they also observed similar issue: such datasets suffered from bound- ary ambiguity and noise frequently, even when the segmentation points were determined by human an- notators. Zhang and Zhou (2019) claimed that they collected 4k manually labeled topic segments and automatically augmented them to be approximately 40k dialogues in total; however, until now, these annotated data are still not publicly available.
Dialseg711
(Xu al., 2021)
X
CSM/doc2dial (Xing and Carenini, 2021)
X
TIAGE
(Xic ct al., 2021)
300
X
CSTS
(Xia ct al., 2022)
506
Role
X
SuperDialseg
Ours
6,863
Role.DA
Table 1: Summary of dialogue segmentation datasets. To the best of our knowledge, SuperDialseg the largest open-sourced supervised dataset for dialogue segmentation. (Ex-Anno: Extra annotations, DA: Di- alogue act)
*These data are augmented by random combinations.
The SuperDialseg Dataset
Documents have natural structures, whereas natu- ral structures for dialogues are hard to define. In this paper, we utilize the well-defined document structure to define the dialogue structure.
3.1 Data Source
Doc2dial (Feng et al., 2020) and MultiDoc2Dial (Feng et al., 2021a) are two popular and high- quality document-groundeer dialogue corpora. Compared to other DGDS datasets (Zhou et al., 2018; Dinan et al., 2019), they contain fine- -in-neri
For evaluation, Xu al. (2021) constructed the Dialseg711 dataset by randomly combining dia- logues in different domains. Automatically gener- ating large-scale datasets in this way seems plausi-

---

grounding references for each utterance, as well additional dialogue-related annotations like di- alogue acts and role information. Based on these annotations, we can construct a supervised dia- logue segmentation dataset without human effort. Detailed information about doc2dial and Multi- Doc2Dial can be found in Appendix B.
had document-level segmentation points, but our definition is more fine-grained and can describe the fine-grained topic shifts in conversations.
Furthermore, we carefully consider some de- tails in the DGDS datasets. In the doc2dial and MultiDoe2Dial datasets, users sometimes ask ques- tions that require agents' further confirmation, like 'Hello. I' I'd like to learn about your retirement pro- gram in Figure However, in this case, though these queries and their following utterances are discussing the same topic, their grounding spans may not locate in the same section, leading to mistakes. Therefore, we corrected it as a non- segmentation point if the dialogue act of utter- ances from users is 'query condition' (the user needs further confirmation). Besides, also moved the dialogues with meaningless utterances like 'null'/'Null to further improve the quality of our supervised dataset.
3.2 Data Construction Approach
Since doc2dial and MultiDoc2Dial provide the grounding lexts for utterances, such that we can un- derstand what the utterances are about according to the grounding references documents. Take Fig- ure I as an example. The user query 'Will my child receive the full payment up front or monthly pay- ments' refers to the information from the section Beeefiit For Your Family'. Then, the agent reads the document, sceking information from the same section, and provides proper response. In this case, we know they are discussing same topic be- cause the subject of these utterances locates in the same section of the document. During this interac- tion in the DGDS scenario, we find that it provides an alignment between documents and dialogues. In this manner, we can define the segmentation points of dialogue based on the inherent structure of docu- ments considering the grounding reference of each utterance. Furthermore, the grounding reference can reflect what interlocutors are thinking about when talking, which precisely matches the defini- tion discussed in Section Thus, this validates the rationality of our data collection method. In this pa- per, we focus on segmenting this type of dialogue, which is also dominant in recent ChatGPT-like dia- logue systems (c.g., ChatPDF)). Surprisingly, our empirical studies indicate that models trained on our datasets can also perform well other types of dialogue (i.e. wwwrdduuuuntggrouuudee dialogue). We will discuss later in Section 5.4.
Moreover, the dev/test sets of the DGDS datasets contain some unseen documents with respect to the training set. Consequently, some dialogues in the dev/test sets discuss unseen topics with respect to the training set. Therefore, following the offi- cial splitting can serve as an excellent testbed to develop dialogue segmentation models with high generalization ability. Algorithm shows the pro- cedure of determining the segmentation points for document-groundde dialogue. Note that the is set containing single/multiple grounding refer- ence(s).
A Algorithm 1 Segment dialogue
Require: U Uu J utterances
Require: G= {g))) grounding spans
Require: Sec(g)) return the section IDs
Require: R = [ri] roles
Require: DA = {do:) dialogue acts
for i 1.... do
Based on this idea, we determine that one ut- terance starts new topic when its grounding ref- erence originates from different section in the grounded document compared to its previous utter- ance. Besides, there may be multiple grounding references for one utterance. In such a case, if all references of the current utterance are not located in the same section of any reference of the previous utterance, we regard it as the beginning of a new topic. Otherwise, these two utterances refer to the same topic. Note that MultiDocDiaal has alrcady
if Sec(g)) Sec(g; 1) 0 then
I(u4-1) =1
4: end
5: end for
6: for i = 0.--.,N. N do
7: if da; = query condition and r;
user then L(w.) = 0
end if
10: end for
11: L(ux)
12: return L: List of segmentation points.
Phttps: //wwww wwtp:.chttdf.coomm com/

---

Besides, doc2dial and MultiDoc2iaal provide useful dialogue-related annotations about the role (user or agent) and dialogue act of each utterance (e.g., respond solution). We inherit them to en- rich our supervised dialogue segmentation corpus. Intuitively, dialogue segments are characterized by specific patterns of roles and dialogue acts. For example, when users ask questions, the agents will provide answers ask another question to confirm some details. The discussed topies are generally maintained until the problem solved. Therefore, we hypothesize that introducing dialogue act infor- mation is helpful for dialogue segmentation. We conducted a pilot experiment on the relationship between the segmentation points and the dialogue acts, proving our hypothesis. Figure 2 illustrates the distribution of dialogue acts for all utterances and those serve as the segmentation points in the test set of SuperDialseg. We observe that most dialogue acts are highly related to the segmenta- tion points. Specifically, utterances with dialogue acts DA-1, DA-5, DA-6, and DA-7 probably act as segmentation points, which indicates that a topic usually ends when solutions are provided, or the previous query has solution. Therefore, these extra dialogue-related annotations are significant for the task of dialogue segmentation. We also conducted an analysis in Section 5.4.4 to further confirm our hypothesis.
of segment is shown in Appendix C.
#Token/Dial
#Turn/Dial #Segment/Dial
Split #Sample
Aver
Max
Aver
Max
Aver
Max
Train Valid Test
6,863
239
646
4.25
1,305
231
553
4.27
1,310
220
548
20
4.09
9
Table 2: Statistics of our SuperDialseg dataset.
3.3 Human Verification
To verify the quality of our proposed dataset, we conducted human verification to confirm the ratio- nality of our data collection method. We randomly selected 100 dialogues from the test set of Super- Dialseg and each dialogue was annotated by two human annotators. We collected the annotations from the crowd-soureing platform, Amazon Me- chanical Turk and 20 valid workers participated in our annotation task. Each assignment contains 10 dialogues to be annotated.
Since it is a verification process, it should be con- ducted more carefully than collecting data for train- ing. Therefore, we only allowed workers whose HIT approval rate is greater than 95% to access and also rejected some invalid results, for example, in- cluding too many consecutive segmentation points. Detailed rejection rules and the user interface of our verification task can be found in Appendix D. For each dialogue, we asked two unique workers to do the annotation and calculated the Cohen's 's Kappa scores between their annotations and the labels constructed by our data collection method. Finally, the Kappa score is 0.536, which is higher than TIAGE's Kappa score. Therefore, we can conclude that SuperDialseg has similar or even higher quality than the manually collected corpus (i.c., TIAGE) so that it can serve as a good train- ing resource and testhed for the task of dialogue segmentation. Note that our automatic data col- lection method does not require human effort and can be easily scaled up long as extra annotated document-grounded dialogue corpora available.
Dialogue act distribution of SuperDialseg
0.0%
6K
75.2%
#DA SegPoint
#DA In dataset
query
#Utterance
4K
query. solution
respond solution
respanse pasitive
DA response nogative
3K
DA-5: respond noo ution
0.3%
DA-6: respond, solution, positive
7: respond solution on, negative
2K
1.7%
42.2%
1K
77.5%
78.6%
90.0%
0
DA-0
DA-1
DA-2
DA-3
DA-4
DA-5
DA-6
DA-7
Dialogue act
Figure 2: Distribution of dialogue acts for utterances and for those at the segmentation points in the test set of SuperDialseg. The percentages represent the pro- portion of utterances where they serve as segmentation points among all utterances belonging the same dia- logue act.
Task definition
Dialogue Segmentation aims to segment a dialogue D = [U1, U2, Un} into several pieces accord- ing to their discussed topics, where U, contains not only the utterance u; but also some proper- ties including the role r; and the dialogue act da;.
Table summarizes the statistics of the Super- Dialseg dataset. Detailed distribution of the number
"ht / http:://wwwwwttwtt// nttur.ccom

---

With the SuperDialseg dataset, we define it as supervised learning task by labeling each utterance with it is the end of a segment, otherwise, y =.
TextTiling+[CLS] (Xu et al., 2021) further en- hanced TexcTiling with [CLS] representation from pre-trained BERT (Devlin et al., 2018). Xu et al. (2021) improved the TextTiling algorithm to be the GreedySeg algorithm to provide more accurate di- alogue segments for response selection. Xing and Carenini (2021) designed TextTiling+ NSP, which uses the next sentence prediction (NSP) score of BERT as the similarity between adjacent utter- ances to enhance Tex(Tiling, and also pre-trained trainraiedd coherence scoring model (CSM) on large-scale dialogue corpora with 553k samples to compute the coherence score of two adjacent utterances instead. Koshorek et al. (2018) proposed an LSTM-based supervised text segmentation model called TextSeg. Since it is trained on the Wiki727K dataset without using annotated dialogue segmentation datasets, we regard it as an unsupervised method for dialogue segmentation, denoted TextSegient-
5 Experiments
5.1 Datasets
SuperDialseg is our proposed dataset, containing 9,478 conversations with rich annotations. Details are provided in Section 3.
TIAGE (Xie et al., 2021) is a manually anno- tated dataset extended from PersonaChat (Zhang et al., 2018) comprising 500 samples, where 300, 100, and 100 samples are for the training, testing, and validation sets, respectively.
DialSeg711 (Xu al., 2021) is an evalua- tion dataset containing 711 multi-turn conversa- tions, which are synthesized by combining single- topic dialogues sampled from the MultWOOZ (Budzianowski et al., 2018) and the Stanford Dia- logue Dataset (Eric et al., 2017).
LLM-based Methods. Recently, LLMs achieve remarkable success in various fields. We investi- gate how well they segment dialogues by ap- plying the InstructGPT (Ouyang et al., 2022) and ChatGPT based on simple prompt templates.
5.2 Comparison Methods
Supervised DL-based Methods. We design some strong baselines based on pre-trained lan- guage models (PLMs), including BERT, RoBERTa (Liu et al., 2019a), and TOD-BERT (Wu al., 2020). Besides, we train TextSeg but using Super- Dialseg or TIAGE dataset, denoted as TextSegdial- Xie et al. (2021) proposed RetroTS-T5, which a generative model for dialogue segmentation.
We construct an extensive benchmark involving 18 models across five categories. including random methods, unsupervised traditional methods, unsu- pervised DL-based methods, LLM-based methods, and supervised DL--based methods.
Random Methods. We design two random methods to show the bottom bound of the dia- logue segmentation task. In an n-turn dialogue, we randomly select the number of segmentation points from {O..... places segmentation point with the probability c n whereas Even distributes the segmentation bound- aries evenly based on the interval [_].
We try our best reproduce the reported per- formances in published papers with official open- sourced resources and our implementation. Most of the results are similar or even superior to those reported. Further details about all the methods and the implementation are provided in Appendix A for reproducing the results in our benchmark.
1}. Then, Random
Unsupervised Traditional Methods. We se- lect three representative algorithms in this cate- gory. BayesSeg (Lisenstein and Barzilay, 2008) is Bayesian lexical cohesion segmenter. GraphSeg (Glavaš et al., 2016) segments by solving maximum clique problem of a semantic association graph with utterance nodes. Tex(Tiling (Hearst, 1997) determines the segmentation point when the calcu- lated depth score of pair of adjacent utterances is beyond given threshold.
5.3 Evaluation Metrics
To evaluate the model performances, we use the following metrics: (1) Pæ-error (Beeferman et al., 1999) checks whether the existences of segmen- tation points within sliding window in predic- tions and references coincident; (2) WindowD- iff (WD) (Pevzner and Hearst, 2002) is similar Pg. but checks whether the numbers of segmenta- tion points within a sliding window in predictions and references are the same; P3 and WD compute soft errors of segmentation points within sliding
Unsupervised DL-based Methods. Song al. (2016) enhanced TextTiling by word embeddings. We use Glove embedding (Pennington et al., 2014) here and denote as Gexooe

---

SuperDialseg
TIAGE
Dinlseg711
Model
Pk]
WD
FIf
MAEL Score]
Pk]
WDJ
FIt
MAET
Scoret
Pk]
WD.
FIf
MAEL
Score?
Without Any Amuotated Corpus
Random
0.494
0.649
0.266
3.819
0.347
0.526
0.664
0.237
4.793
0.321
0.533
0.714
0.204
9.760
0.2%0
Even
0.471
0.641
0.308
4,075
0.376
0.489
0.650
0.241
5.095
0.336
0.502
0.690
0.269
10.061
0.337
BayesSeg
0.433
0.593
0.438
2.934
0.463
0.486
0,571
0.366
2,870
0.419
0.306
0,350
0.556
2.127
0.614
TexrTiling
0.441
0.453
0.388
1.056
0.471
0.469
0.488
0.204
1.420
0.363
0.470
0.493
0.245
1,550
0.382
GrapbScg
0.450
0.454
0.249
1.721
0.398
0.496
0.515
0.238
1,430
0.366
0.412
0.442
0.392
1,900
0.483
Cexooiing-Clove
0.519
0.524
0.353
1.192
0.416
0.486
0.511
0.236
1.350
0.369
0,399
0.438
0.436
1,948
0.509
Tex Tex(Tilin-((CLS)
0.493
0.523
0.277
1.195
0.385
0.521
0,556
0.218
1,400
0.340
0.419
0.473
0.351
2.236
0.453
Greedy Seg
0.490
0.494
0.365
1,490
0.437
0,490
0.506
0.181
1.520
0.341
0.381
0.410
0.445
1.419
0.525
TexTTilngg
0.512
0.521
0.208
1.747
0.346
0.425
0.439
0.285
1.800
0.426
0.347
0.360
0.347
1,969
0.497
CSM
0.462
0.467
0.381
1.261
0.458
0.400
0.420
0.427
1.370
0.509
0.278
0.302
0.610
1,450
0.660
TextSggooe
0.838
2.867
0.113
0.852
0,855
3.090
0.076
0.945
0.036
InstructGiPT
0.365
0.377
0.529
1.255
0.579
0.510
0,552
0.280
1,930
0.375
0.410
0.465
0.515
3,838
0.539
CharGPT
0.318
0.347
0.658
1.610
0. 0.663
0.496
0,560
0.362
2.670
0.417
0.290
0.355
0.690
3.423
0.684
Supervised Learning TIAGE
TextSegahai
0.552
0.570
0.199
1.672
0.319
0.357
0.386
0.450
1.339
0.539
0.476
0.491
0.182
1.794
0.349
BERT
0.511
0,513
0.043
2,955
0.266
0.418
0.435
0.124
2.520
0.349
0.441
0.411
0.005
3.815
0.297
RoBERTa
0.436
0.276
2.262
0.420
0.265
0.287
0.572
1,206
0.648
0.197
0.210
0.650
1.325
0.723
TOD-BERT
0.505
0.514
0.087
2.447
0.289
0,449
0.469
0.128
2.071
0.335
0.437
0.463
0.072
3,708
0.311
R-trTSS-T
0.504
0.505
0.095
2.867
0.295
0.280
0,317
0.576
1,260
0.639
0.331
0.352
0.303
1.442
0.481
Supervised Learning on SuperDiabeg
TextSegaat
0.199
0.204
0.760
0.813
0.779
0.489
0.508
0.266
1.678
0.384
0.453
0.461
0.367
3.387
0.455
BERT
0.214
0.225
0.725
0.934
0.753
0.492
0.526
0.226
L.639
0.359
0.401
0.473
0.381
2.921
0.472
RoBERTS
0.185'
0.192'
0.784
0.816
0.798'
0.401
0.418
0.373
1.495
0.482
0.241
0.272
0.660
1,460
0.702
TOD-BERT
0.220
0.233
0.740
0.910
0.757
0.505
0.537
0.256
1.622
0.367
0.361
0.454
0.534
3.438
0.563
RetroTS-
0.237
0.733
1.005
0.751
0.415
0.439
1,500
0.463
0.321
0.378
2.041
0.572
Table 3: Performances on three datasets. Models in the first part do not access any annotated corpus. Models in the second part are trained on the TIAGE dataset and evaluated on the testing set of SuperDialseg, TIAGE, and Dialseg71 I. Models the third part are similar to the second part's models but trained on SuperDialseg. The best and second- bestt performances are highlighted in bold and underlined, respectively. Different superscripts represent different significance levels (f: p < 0.001 (***), 4: < 0.01 (**)).
windows, and lower scores indicate better perfor- mances. (3) F1 score; (4) mean absolute error (MAE) calculates the absolute difference between the numbers of segmentation points in predictions and references; (5) By considering the soft errors and the F1 score simultaneously, we use the fol- lowing score for convenient comparison:
Carenini (2021), we use F1 (binary) instead of F1 (macro) in the evaluation, because we only care about the segmentation points.
5,4 Experimental Results and Analysis
We conduct extensive experiments demonstrate the effectiveness of our proposed dataset and reveal valuable insights of dialogue segmentation. Table shows the results of our benchmark.
Score 2 Fl PA) (1 WD)
suggested by the ICASSP2023 General Meeting Understanding and Generation Challenge (MUG).
5.4.1 Supervised vs. Unsupervised
For the Pa and WD, previous works adopted dif- ferent lengths of sliding windows (Xu et al. (2021) used four, but Xing and Carenini (2021) used dif- ferent settings), leading to an unfair comparison. Following Fournier (2013), we set the length of the sliding window as half of the average length of segments in dialogue, which is more reason- able because each dialogue has different aver- age length of segments. In contrast to Xing and
When trained on SuperDialseg, supervised meth- ods outperform all unsupervised methods on Super- Dialseg by large margins, and they surpass most of the unsupervised methods on TIAGE when trained on TIAGE, which highlights the crucial role of su- pervised signals in dialogue segmentation. Despite the domain gap, supervised methods exhibit strong generalization ability on out-of-domain data. For example, when trained on SuperDialseg. RoBERTa surpasses all unsupervised methods on Dialseg711 and all except CSM on TIAGE.
'https: //2023. ieeeicassp. org/ si signal-processing -aaii--rooessing-g--------- chaleeeellegg//

---

SuperDiabeg
TIAGE
Dialseg711
Model
Train
Pk.
WDJ
F11
Score"
Pk.
WDJ
F17
Score
Pk.
WDJ
F17
Score
BERT
0.511
0,513
0.043
0.266
0.418'
0.435'
0.124
0.349
0.441
0,411
0.005
0.297
TIAGE
0.505
0.469
0.128
0.072
BERT TOD:BERT
0,297'
0.316
0.622
0.658
0.626
0.278
0.460
0.326
0.406
SuperDialeg 300
0.370
0,392
0.541
0,580
0,541
0,643
0.265
0.336
0.480
0.622
0,349
0.399
Table 4: Comparison at the same scale of training data. The best performances are highlighted in bold. Different superscripts represent different significance levels (f: p < 0.001 (***), :: p 0.01 (**)).
5.4.2 SuperDialseg vs. TIAGE
logue segmentation. For example, documents gen- crally discuss a topic with many long sentences, whereas topies in conversations generally involve fewer relatively shorter utterances. Therefore, even though have large-scale text segmentation cor- pora, learning only on documents not sufficient for segmenting dialogues, which convinces the ne- cessity of constructing large-scale supervised seg- mentation datasets specially for dialogues.
Although TIAGE is the first open-sourced super- vised dataset for dialogue segmentation, only has 300 training samples, which may not be suf- ficient for deep learning models. Our experi- ments show that trained on TIAGE, BERT and TOD-BERT perform poorly on Dialseg711 with low Fl scores, while they even perform worse than random methods on SuperDialseg. Addi- tionally, we observe that when trained on TIAGE, RetroTS-T5 usually generates strange tokens such as "<extra_id_>> I' and 'Doar pentru', rather tive'. These observations reflect that small training sets generally result in unstable training processes.
5.4.4 Effect of Dialogue-related Annotations
SuperDialseg provides additional dialogue-related annotations such as role information and dialogue acts. We conduct analysis to show their effect on dialogue segmentation. We introduce role in- formation and dialogue act by adding trainable em- beddings to the inputs, denoted as the Multi-view RoBERTa (MVRoBERTa). Table 5 shows that MVRoBERTa outperforms the vanilla RoBERTa, which confirms that considering dialogue-related annotations is important for dialogue segmentation.
than
Besides, we try to conduct comparison between SuperDialseg and TIAGE on the same scale. We randomly selected 300/100 training/validation sam- ples from SuperDialseg, denoted as SuperDialseg- 300. Table 4 shows that though the performances are worse than those trained with the full train- ing set, training with the subset of SuperDialseg is still more stable and performs relatively well on all datasets. We believe it is because SuperDialseg has more diverse patterns for dialogue segmentation. Furthermore, except for RoBERTa, other models trained on SuperDialseg outperform those trained on TIAGE when evaluating on Dialseg711 which proves the superiority of our proposed dataset.
SuperDialseg
Model
Pk]
F11
MAEJ
Scoret
ROBERBa MVRoBERTA
0.185
0.192
0.784
0.816
0.798
0.178
0.185
0.797
0.792
0.808'
Table dialogue-related annotations. (#:p 0.05 (+))
Experiment for exploring the effect of
5.4.5 Other Valuable Findings
5.4.3 DialogueSeg vs. TextSeg
CSM is strong unsupervised method, as it achieves the best performance on TIAGE and Di- alseg711 compared with other unsupervised meth- ods, LLM-based methods, and supervised methods, demonstrating the potential of unsupervised ap- proaches in this task. However, it does not perform consistently well on SuperDialseg. As previously discussed in Section 5.1, utterances in Dialseg711 are incoherent when topies shift, and dialogues in TIAGE 'rush into changing topics' as stated in Xie et al. (2021). Therefore, CSM, which mainly learns sentence coherence features and coarse topic
It may seem plausible that models trained on large amounts of encyclopedic documents can be used to segment dialogues if we treat them as gen- eral texts. However, our results show that even though TextSegrezt is trained with more than 100 times (w.rt. SuperDialseg) and even 2,400 times (w.rt. TIAGE) more data than TextSegdial it per- forms significantly worse than TextSegdial on all evaluation datasets. We attribute this to the fact that TextSegreat neglects the special characteris- ties of dialogues, and thus, unsuitable for dia-

---

information, fails segment real-world conver- sations with fine-grained topics. This observa- tion strengthens the necessity of our SuperDialseg dataset for evaluating dialogue segmentation mod- els properly. Surprisingly, ChatGPT, despite using a simple prompt, achieves the best performance on SuperDialseg and Dialseg711 compared with other methods without accessing annotated corpora. Moreover, LLM-based methods perform compet- itively on TIAGE. We believe with better prompt engineering, they can serve as stronger baselines. Additionally, BayesSeg performs relatively well compared to some DL-based methods, showing the importance of topic modeling in this task. We highlight this as future direction in this area.
Limitations
In this study, we notice that our proposed method of introducing dialogue-related information in- cluding role information and dialogue act (i.c., MVRoBERTa) performs well but the way of ex- ploiting these annotations is simple. We believe there is much room for improvement. Therefore, we need find a more proper and effective way make full use of the dialogue- information, expanding the potential applications of supervised dialogue segmentation models.
Besides, our dataset is only available for train- ing and evaluating English dialogue segmentation models. Recently, Fu et al. (2022) proposed a Chi- nese DGDS dataset with fine-grate fine-grained grounding reference annotations, which can also be used construct another supervised dataset for Chinese di- alogue segmentation. We leave as future work.
6 Conclusions
In this paper, we addressed the necessities and chal- lenges of building large-scale supervised dataset for dialogue segmentation. Based on the charac- teristics of DGDS, we constructed large-scale supervised dataset called SuperDialseg, also inher- iting rich annotations such as role information and dialogue act. Moreover, we confirmed the quality of our automatically constructed dataset through human verification. We then provided an exten- sive benchmark including 18 models across five categories on three dialogue segmentation datasets, including SuperDialseg, TIAGE, and Dialseg71 1.
Ethies Statement
In this paper, we ensured that we adhered to ethical standards by thoroughly reviewing the copyright of two involved DGDS datasets, the doc2dial and the MultiDoe2Dial. We constructed the SuperDialseg dataset in compliance with their licenses (Apache License 2.0). We took care of the rights of the creators and publishers of these datasets and added citations properly.
As for the process of human verification, we collected the data from Amazon Mechanical Turk. Each assignment cost the workers 20 minutes on average. Each worker was well paid with $2.5 af- ter completing one assignment, namely, the hourly wage is $7.5, which is higher than the federal mini- mum hourly wage in the U.S. ($7.25).
To evidence the significance and superiority of our proposed supervised dataset, we conducted extensive analysis in various aspects. Empirical studies showed that supervised signals are essen- tial for dialogue segmentation and models trained SuperDialseg can also be generalized out-of- domain datasets. Based on the comparison between the existing dataset (i.e. TIAGE) and SuperDialseg on the same scale, we observed that models trained on SuperDialseg had a steady training process and performed well on all datasets, probably due to its diversity in dialogue segmentation patterns, there- fore, it is suitable for this task. Furthermore, the comparison between RoBERTa and MVRoBERTa highlighted the importance of dialogue-related an- notations previously overlooked in this task.
Acknowledgements
This work was supported by JST SPRING, Grant Number JPMISP2108.
References
Sebastian Arnold, Rudolf Schneider, Philippe Cudré- Mauroux, Felix A. Gers, and Alexander Löser. 2019. SECTOR: A neural model for coherent topic seg- mentation and classification. Transactions of the As- sociation for Computational Linguistics, 7:169-184
Additionally, we also provided some insights for dialogue segmentation, helping the future develop- ment in this field. Moreover, dataset and codes are publicly available support future research.
Joe Barrow, Rajiv Jain, Vlad Morariu. Varun Manju- natha, Douglas Oard, and Philip Resnik. 2020. A joint model for document segmentation and segment

---

labeling, In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguis- rics, pages 313-322, Online. Association for Com- putational Linguistics.
Jacob Eisenstein and Regina Barzilay, 2008. Bayesian unsupervised topic segmentation. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, EMNLP '08, page 334-343, USA. Association for Computational Linguistics.
Doug Beeferman, Adam Berger, and John Lafferty. 1999. Statistical models for text segmentation. Ma- chine learning, 34(1)2177-
Mihail Eric, Lakshmi Krishnan, Francois Charette, and Christopher D. Manning. 2017. Key-value retrieval networks for task-oriented dialogue. In Proceedings of the 18th Annual SIGdial Meeting on Discourse and Dialogue, pages 37-49, Saarbrücken, Germany. Association for Computational Linguisties.
Narjes Boufaden, Guy Lapalme, and Yoshua Bengio. 2001. Topic segmentation A first stage to dialog- based information extraction. In NLPRS.
Gillian Brown and George Yule. 1983, Discourse anal- ysis. Cambridge university press.
Song Feng, Siva Sankalp Patel, Hui Wan, and Sachin- dra Joshi. 2021a. MultiDoc2Dial: Modeling dia- logues grounded in multiple documents. Proceed- ings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6162 6176. Online and Punta Cana, Dominican Republic, Asso- ciation for Computational Linguistics.
Pawel Budzianowski, "sinn Wen, Bo-Hsiang Tseng, Iñiio Casanueva, Stefan Ultes, Osman Ra- madan, and Milica Gašić. 2018. MultiWOZ. - a large-scale multi-domain Wizard-of-Oa -z dataset for task-oriented dialogue modelling. Proceedings of the 2018 Conference on Empirical Methods in Nat- ural Language Processing, pages 5016-5022 Brus- sels, Belgium. Association for Computational Lin- guistics.
Song Feng, Hui Wan, Chulaka Gunasekara, Siva Patel, Sachindra Joshi, and Luis Lastras. 2020. doc2dial: A goal-oriented document-groundee logue dataset. In Proceedings of the 2020 Confer- ence Empirical Methods in Natural Language Processing (EMNLPP), pages 81 18-8128, Online, As- sociation for Computational Linguistics.
dia-
Jiaao Chen and Diyi Yang. 2020. Multi-view sequence- se-sequence models with conversational structure for abstractive dialogue summarization. In Proceed- ings of the 2020 Conference Empirical Methods Natural Language Processing (EMNLP), pages 4 4106-4118, 18, Online. Association for Computational Linguistics.
Xiachong Feng, Xiaocheng Feng, Libo Qin. Bing Qin, and Ting Liu. 202Ib. Language model as an an- notator: Exploring DialoGPT for dialogue summa- rization. Proceedings of the 59th Annual Meet- ing of the Association for Computational Linguistics and the 11th International Joint Conference Nat- ural Language Processing (Volume 1: Long Papers), pages 1479-1491, Online. Association for Computa- tional I. Linguistics.
Freddy Y. Y. Choi. 2000. Advances in domain inde- pendent linear text segmentation. In 1st Meeting of the North American Chapter of the Association for Computational Linguisties.
Chris Fournier. 2013. Evaluating text segmentation us- ing boundary edit distance. In Proceedings of the 51st Aunual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), pages 1702 1712, Sofia, Bulgaria. Association for Compu- tational Linguistics.
Arman Cohan, Iz Beltugy, Daniel King, Bhavana Dalvi, and Dan Weld. 2019. Pretrained language models for sequential sentence classification. In Proceed- ings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th Inter- national Joint Conference on Natural Language Pro- cessing (EMNLP-IJCNLP) pages 3693-3699, Hong Kong. China. Association for Computational Lin- guistics.
Haomin Fu, Yeqin Zhang, Haiyang Yu, Jian Sun, Fei Huang, Luo Si, Yongbin Li, and Cam Tu Nguyen. 2022. Doc2Bot: Accessing heterogeneous docu- ments via conversational bots. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 1820- 1836, Abu Dhabi, United Arab Emirates. Association for Computational Linguis- tics.
Abhishek Das, Satwik Kottur, Khushi Gupta, Avi Singh, Deshraj Yadav, José MF Moura, Devi Parikh, and Dhruv Batra. 2017. Visual dialog. In Proceed- ings of the IEEE conference on computer vision and pattern recognition, pages 326-335.
Haoyu Gao. Rui Wang. Ting-En Lin, Yuchuan Wu. Min Yang. Fei Huang, and Yongbin Li. 2023. Un- supervised dialogue topic segmentation with topic- aware contrastive learning. In Proceedings of the 46th International ACM SIGIR Conference on Re- search and Development in Information Retrieval, pages 2481-2488
Jacob Devlin, Ming-We Wei Chang, Kenton Lee, and Kristina Toutanova. 2018 Bert: Pre-training of deep bidirectional transformers for language understand- ing. Xiviv preprint 1810.08110004000
Emily Dinan, Stephen Roller, Kurt Shuster, Angela Fan, Michael Auli, and Jason Weston. 2019. Wizard of wikipedia: Knowledge-powers conversational agents. In International Conference on Learning Representations.
Jeroen Geertzen and Harry Bunt. 2006. Measuring annotator agreement in a complex hierarchical dia- logue act annotation scheme. In Proceedings of the

---

7th SIGdial Workshop on Discourse and Dialogue, pages 126-133. Sydney, Australia, Association for Computational Linguistics.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hin- ton. 2017. Imagenet classification with deep con- volutional neural networks. Communications of the ACM, 60(6):4-90.
Robert Geirhos, fömn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann. 2020. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2(1 1):6656673
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy. Mike Lewis. Luke Zettlemoyer, and Veselin Stoyanov. 2019a. Roberta: A robustly optimized bert pretraining ap- proach. arXiv preprint 1rXiv:1907.1109:
Goran Glavaš, Federico Nanni, and Simone Paolo Ponzetto. 2016. Unsupervised text segmentation us- ing semantic relatedness graphs. In Proceedings of the Fifth Joint Conference on Lexical and Computa- tional Semanties, pages 125 130. Berlin, Germany. Association for Computational Linguistics.
Zhengyuan Liu, Angela Ng. Sheldon Lee, Ai Ti Aw, and Nancy Chen. 2019b. Topic-aware pointer- generator networks for summarizing spoken conver- sations. In 2019 IEEE Automatic Speech Recogni- tion and Understanding Workshop (ASRU), pages 814 821. IEEE
Goran Glavaš and Swapna Somasundaran. 2020. Two- level transformer and auxiliary coherence model- ing for improved text segmentation. Proceedings of the AAAI Conference on Artificial Intelligence, 34(05):7797-7804.
Ilya Loshchilov and Frank Hutter. 2018. Decoupled weight decay regularization. In International Con- ference on Learning Representations.
Zheng Gong, Shiwei Tong, Han Wu, Qi Liu, Hanqing Tao, Wei Huang, and Runlong Yu. 2022. Tipster: topic-guided language model for topic-aware text segmentation. In Database Systems for Advanced Applications: 27th International Conference, DAS- FAA 2022, Virtual Event, April 11-14, 2022, Pro- ceedings, Part III, pages 213-221..
Longxuan Ma. Wei-Nan Zhang, Mingda Li, and Ting Liu. 2020, survey of document grounded dialogue systems (dgds). arXiv preprint arXiv:2004. 13818.
1. McCowan, J. Carletta, W. Kraaij. S. Ashby, S. Bour- ban, M. Flynn, M. Guillemot, T. Hain, J. Kadlec, V. Karaiskos, M. Kronenthal, G. Lathoud, M. Lin- coin, A. Lisowska, W. Post, Dennis Reidsma, and P. Wellner. 2005. The ami meeting corpus. In Pro- ceedings of Measuring Behavior 2005, 5th Interna- tional Conference on Methods and Techniques in Be- havioral Research, pages 137-140. Noldus Informa- tion Technology.
Barbara Grosz and Candace L. Sidner. 1986. Atten- tion, intentions, and the structure of discourse. Com- putational Linguistics, 175-244..
Marti A. Hearst. 1997. Text tiling: Segmenting text subtopic passages. Computa- tional Linguisticss 23(1)333-64.
into
Long Ouyang, Jeff Wa, Xu Jiang, Diogo Almeida, Car- roll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow in- structions with human feedback. arXiv preprint arXiv:2203.02155.
Julia Hirschberg and Diane Litman. 1993. Empirical studies on the disambiguation of cue phrases. Com- putational Linguistics, 19(3):501-50..
Adam Janin, Don Baron, Jane Edwards, Dan Ellis, David Gelbart, Nelson Morgan, Barbara Peskin, Thilo Pfau, Elizabeth Shriberg, Andreas Stolcke. al. 2003. The iesi meeting corpus. In 2003 IEEE International Conference on Acoustics, Speech, and (ICAAd
Rebecca J. Passonneau and Diane J. Litman. 1997.
Discourse segmentation by human and automated means. Computational Linguistics, 23(1):103-139
Signal Processing, 2003, volume pages I-I. IEBF.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. 2019. Pytorch: An imperative style. high high-performance deep learning library. Advances in neural information processing systems, 32.
Jachun Jung, Bokyung Son, and Sungwon Lyu. 2020. AttnOO: Knowledge Graph Exploration with In-and- Out Attention Flow for mhowledgeGGrounder Dia- logue. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Process- ing (EMNLP), pages 3484-3497, Online. Associa- tion for Computational Linguistics.
Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. GloVe: Global vectors for word representation. In Proceedings of the 2014 Confer- ence on Empirical Methods in Natural Language Processing (EMNLP), pages
Omri Koshorek, Adir Cohen, Noam Mor, Michael Rot- man, and Jonathan Berant, 2018. Text segmentation supervised learning task. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, Volume 2 (Short Pa- pers), pages 469-473, New Orleans, Louisiana. As- sociation for Computational Linguistics.
Doha, Qatar. Association for Computational Linguistics.
Lev Pevzner and Marti A. Hearst. 2002. A Critique and Improvement of an Evaluation Metric for Text Seg- mentation. Computational Linguisties, 28(1):19- 36.

---

Stephen Roller, Emily Dinan, Naman Goyal, Da Ju. Mary Williamson, Yinhan Liu, Jing Xu, Myle Ott, Fric Michael Smith, Y-Lan Boureau, and Jason We- ston. 2021. Recipes for building an open-domain chatbot. In Proceedings of the 16th Conference of the European Chapter of the Association for Compu- tational Linguisties: Main Volume, pages 300-325, Online. Association for Computational Linguistics.
Linzi Xing and Giuseppe Carenini. 2021. Improv- ing unsupervised dialogue topic segmentation with utterance-pair coherence scoring. In Proceedings of the 22nd Annual Meeting of the Special Inter- est Group on Discourse and Dialogue, pages 167- 177, Singapore and Online. Association for Compu- tational Linguistics.
Yi Xu, Hai Zhao, and Zhuosheng Zhang. 2021. Topic- aware multi-turn dialogue modeling, Proceedings of the AAAI Conference on Artificial Intelligence. 311761144166114884
Lifeng Shang, Zhengdong Lu, and Hang Li. 2015. Neu- ral responding machine for short-text conversation. In Proceedings of the 53rd Annual Meeting of the A Association for Computational Linguistics and the 7th International Joint Conference on Natural Lan- guage Processing (Volume 1: Long Papers), pages 1577-1586, Beijing, China. Association for Compu- tational Linguistics.
Chenxu Yang. Zheng Lin, Jiangnan Li, Fandong Meng, Weiping Wang, Lanrui Wang, and Jie Zhou. 2022. TAKE: Topic-shift aware knowledge sElection for dialogue generation. In Proceedings of the 29th In- ternational Conference on Computational Linguis- tics, pages 253-265, Gyeongju, Republic of Korea, International Committee on Computational Linguis- tics.
Yiping Song, Lili Mou, Rui Yan, Li Yi, Zinan Zhu, Xi- aohua Hu, and Ming Zhang. 2016. Dialogue session segmentation by embedding enhannced texttiling. In- terspeech 2016, pages 2706-2710.
Leilan Zhang and Qiang Zhou. 2019. Topic segmenta- tion for dialogue stream. In 2019 Asia-Pacify Sig- nal and Information Processing Association Annual Summit and Conference (APSIPA ASC). pages 1036- 1043, IEEE
Ryuichi Takanobu, Minlie Huang, Zhongzhou Zhao, Fenglin Li, Haiqing Chen, Liqiang Nie, and Xi- aoyan Zhu. 2018. weakly supervised method for topic segmentation and labeling in goal-oriented di- alogues via reinforcement learning. In IJCAI-ECAI, pages 4403-4410
Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Szlam, Douwe Kiela, and Jason Weston. 2018. Per- sonalizing dialogue agents: have a dog, do you have pets too? In Proceedings of the 56th An- nual Meeting of the Association for Computational Linguistics (Volume Long Papers), pages 2204- 2213, Melbourne, Australia. Association for Com- putational Linguistics.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Remi Louf, Morgan Funtow- icz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame. Quentin Lhoest, and Alexander Rush. 2020. Trans- formers: Mtate-oftthe-art natural language process- ing. In Proceedings of the 2020 Conference on Em- pirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online, Asso- ciation for Computational Linguistics.
Ming Zhong, Yang Liu, Yichong Xu, Chenguang Zhu, and Michael Zeng. 2022. Dialoglm: Pre-trained model for long dialogue understanding and summa- rization. Proceedings of the AAAI Conference on Artificial Intelligence, 36(10)
Chien-Sheng Wu, Steven C.H. Hoi, Richard Socher, and Caiming Xiong. 2020. TOD-BERT: Pre-trained natural language understanding for task-oriented di- alogue. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 917-929, Online. Association for Computational Linguistics.
Kangyan Zhou, Shrimai Prabhumoye, and Alan W Black. 2018. A dataset for document grounded con- versations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Process- ing, pages 708-714 108-713, Brussels, Belgium. Association for Computational Linguistics.
Jinxiong Xia, Cao Liu, Jiansong Chen, Yuchen Li, Fan Yang, Xunliang Cai, Guanglu Wan, and Houfeng Wang. 2022. Dialogue topic segmentation via paral- lel extraction network with neighbor smoothing. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Infor- mation Retrieval, pages 2126-2131.
A Details for Reproduction
A.1 Comparison Methods
To guarantee a fair comparison in our benchmark, we use the open-sourced codes and pre-traine pre-trained models of the comparison methods as much as possible to evaluate their performances. However, some additional details are required to reproduce their reported performances. Therefore, we con- duct a similar level of hyperparameter tuning for each model to obtain similar or better results than those reported in previously published papers.
Huiyuan Xie, Zhenghao Liu, Chenyan Xiong, Zhiyuan Liu. and Ann Copestake. 2021. TIAGE: A bench- mark for topic-shift aware dialog modeling. In Findings of the Association for Computational Lin- guistics: EMNLP 2021, pages 1684-1690, Punta Cana, Dominican Republic. Association for Compu- tational Linguistics.

---

For the GraphSeg, we set the value of the re- latedness threshold as 0.5 and the minimum seg- mentation sizes as to obtain a better performance than that reported in Xing and Carenini (2021). For Glove, we used the version that was pre-trained with 42 billion tokens of web data from Common Craw19. For GreedySeg and CSM, we corrected some inconsistencies in their open- sourced codes with respect their original pub- lished papers and obtained better performances than their reported results.
ning 10 build the deep learning pipeline in our experiments. The AdamW (Loshchilov and Hutter, 2018) optimizer was used to optimize models' pa- rameters with a learning rate of Ie-5 and a batch size of 8. L22rreullarzzatinn with weight decay of 1e-3 was also applied. We trained our supervised models with 20 epochs for SuperDialseg and 40 epochs for TIAGE and SuperDialseg- 300, respec- tively. We employed early stopping when the cal- culated score (metric (5) in Section 5.3) did not improve for half of the total number of epochs. For RetroTS-T5, we trained the generative T5 for 3 epochs on SuperDialseg and 40 epochs on TIAGE. All experiments were conducted on single A 100 80GB GPU. Except for RetroTS--S, all ex- periments took no longer than hour to finish, whereas training RetroTS-T5 SuperDialseg quired approximately six hours. To reduce experi- mental error, we did experiments in several times and reported the average performances in this pa- per. T-test was also conducted to confirm the sig- nificance of the comparison.
For the result of LLM-based methods, we used the OpenAl API2 with the best model text-davinci-000 for InstructGPT and the recent model gpt-3. 5-5t-trboooo666 3 for ChatGPT in June 2023. The maximum number of completion tokens was 512, and the temperature was set to We used the definition from our paper as described in Section as the task instruction, together with the dialogue input, and an output example specifying the output format to form the input prompt. Table 6 shows our constructed prompt templates for the dialogue segmentation task.
B Detailed Information for Data Source
For the PL.M-based supervised methods, we ex- perimented with the hierarchical architectures but
In real-world applications, the demand for guiding end users to seek relevant information and com- plete tasks is rapidly increasing. Feng et al. (2020) proposed goal-oriented document-grounded dia- logue dataset, called doc2dial, to help this line of re- search. To avoid additional noise from the post-hoc human annotations of dialogue data (Geertzen and Bunt, 2006), they designed the dialogue flows in advance. A dialogue flow is a series of interactions between agents and users, where each turn contains dialogue scene that consists of dialogue acts, role information, and grounding contents from a given document. Specifically, by varying these three fac- tors that are further constrained by the semantic graph extracted from documents and dialogue his- tory, they dynamically built diverse dialogue flows turn-by-turn. Then, they asked crowdsourcing an- notators to create the utterances for cach dialogue scene within dialogue flow. It should be noted that although the dialogue flows were generated automatically, the crowdsourcing annotators would reject some unnatural dialogue flows if they found any dialogue turn that was not feasible to write an utterance. Moreover, each dialogue was created based on a unique dialogue flow.
did not achieve satisfactory results. Therefore, we followed the framework of Cohan et al. (2019) to concatenate all the utterances as long sequence input and use the [SEP]<</>> tokens at the end of each utterance except for TOD-BERT to per- form the classification. Note that there are two separate tokens 
 in the original implementa- tion of RoBERTa, we use the first </<> token to do classification. For TOD-BERT, each utterance has a special token representing its role, so we use the special role tokens [sys] and [usr] to perform classification. The maximum number of tokens in utterance is 25. The size of the utterance-level sliding window during training and inference is
[ 20, and the stride 1.
Other implementations that are not specified fol- low the official codes and their papers with default hyperparameters.
A.2 Implementation Details
We implemented all the models using PyTorch (Paszke al., 2019) and downloaded model weights of PLMs from huggingface Transformers (Wolf et al., 2020). We also utilized Pytorch Light-
Usually, goal-oreeg inoemaationseeeiinn con-
stanford. wtwefooddddd//ooooooee/// ccb/projject//lloee Phttps://openad com/cm///////o api/
"nttps://pytorcchligg lightning readthedocs.

---

versations involved multiple documents. Based on the doc2dial dataset, Feng et al. (2021a) ran- domly combined sub-dialogues from doc2dial which ground on different documents to construct the MultiDoc2Dial dataset. Therefore, each ut- terance only grounds on single document. To maintain the natural dialogue flow after the com- bination, they only split sub-dialogue after an agent turn with dialogue an answer' while the next turn is not 'asking a follow-up question'. Besides, they also re- cruited crowdsourcing annotators to rewrite some unnatural utterances the original utterances from doc2dial are inappropriate after combination.
is greater than 95% can access our task, which ensures the quality of our recruited annotators.
However, we still observed some bad annotations. Therefore, we rejected those assignments if they satisfied any of the following conditions:
The last utterance was not annotated as seg- mentation point, indicating that the annotator did not read our given instructions carefully did not fully understand task.
'responding wi th
The 'N/'' utterance was annotated as seg- mentation point, indicating that the annotator did read our given instructions carefully.
Further details about the data collection methods for doc2dial and MultiDoc2Dial can be found in the original papers (Feng al., 2020, 2021a).
The example modified, indicating that the annotator did not read our given instructions carefully.
Detailed Information of SuperDialseg
An annotation had more than three consecu- tive segmentation points, which is impossible in our cases.
Figure 3 shows the detailed distribution of the num- ber of segments in the test set of SuperDialseg.
All the annotations remained in default.
Distribution of #Segment
350
Besides, we rejected those workers and would not approve any annotations from them ever if they satisfied any the following conditions, because they were probably spam users:
300
250
#Dialogue
200
One submitted an assignment that had more than 10 consecutive segmentation points.
100
One submitted an assignment whose all the annotations remained in default.
50
We hope that practical experience in collect- ing verification data can also be helpful for future rescarchers who want collect annotated dialogue segmentation data by crowdsourcing.
#Segment
Figure 3: Distribution #Segment test set of Super-
Dialseg.
D Detailed Information about Human Verification
Figure 4 shows the user interface of our human ver- ification task. We randomly sampled 100 samples from the test set of SuperDialseg for human veri- fication. For the sake of convenience in designing the user interface, we only selected those dialogues with less than 15 utterances.
We set up a series of constraints to ensure the quality of collected human verification data. First all, we set up a constraint that only those workers whose HIT approval rate for all requesters' HITs

---

Dedogue dngrestaion
Arperter
Telo vollet
Euedeer 2iee
Appenal fame DC Expameer
Lrabrctioss
Shotcate
X
Copmine comerntiin bemetio un
>> Task: Dialogue Segmentation
Annotation Instruction
Mateet sleae SHOUT Calani ensyeertiiin por
Club adbedd
at segnent setten Matwinn stocr to i2 momotte point onhonese uot kove on
>> Conversations (You don't need to do anything on the N/A utterance)
Example
Wocrett DO NOT modlly thes part lu redivesoon
plu
choud tiik family sould heed!! Soc Securty fumbly have sadd through
what benalts rocka??
ogont They survirr bendituu"'ve provide boutt hulf parerts and angitl sotiomint bonetta nighor bonelt get ta sunivona fomyyuu secied
Nete: Pleaes regisent the lowing dategus sased on the intructionn and the euampll sbows.
partage
you
Figure 4: User interface for human verification task on Amazon Mechanical Turk.

---

Prompt template for InstructGPT (text-davinci-003)
Task Instruction
Dialogue Segmentation aims segment dialogue {U1, U2, Un) into
several parts according to their discussed topics. Input
Here is the given dialogue D:
U1: [Utterance 1]
U2: [Utterance 2]
Un: [Utterance n]
Output format
Segment into several parts according to their discussing topics. Output format: Part i: Ua-Ub
Output example:
Part 1: U1-U4
Part 2: U5-U6
Output
Output of the dialogue segmentation task:
Prompt template for ChatGPT
# System message
You are helpful assistance to segment give dialogues.
Please follow the output format. DO NOT explain.
Task Instruction
Dialogue Segmentation aims segment a dialogue {U1, U2, Un) into several parts according to their discussed topics. Please help me to segment the following dialogue: # Input
U1: [Utterance 1]
U2: [Utterance 2]
Un: [Utterance nl
Output format
Segment into several parts according to their discussing topics.
Output format: Part i: Ua-Ub
Output example:
Part 1: U1-U4
Part 2: U5-U6
Output
Output of the dialogue segmentation task:
Table 6: Prompt templates for dialogue segmentation task.


Reviews:
[{"review_id": "IeO6F9bUiV", "reviewer": "Reviewer_pN2F", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your interest in our work! We also hope that SuperDialseg can contribute to the NLP community. And we believe that the following explanation will help to alleviate some of your concerns.\n\nWe agree that the quality of the original DGDS corpora will affect the quality of SuperDialseg. But we would like to point out that the two DGDS datasets we chose are still of high quality and were used as shared tasks held in ACL 2021 and ACL 2022 for evaluating many document-grounded dialogue systems. Here are the home pages of these two datasets (https://doc2dial.github.io/ ; https://doc2dial.github.io/multidoc2dial/) respectively for your reference. \n\nAs for some unexpected errors in these datasets, we also conducted multiple strategies to resolve this issue. For example, as stated in Lines 293-295, we removed the whole dialogue if there were meaningless utterances. We also corrected segmentation errors based on their dialogue act annotations, as stated in Lines 290-293. With these efforts, we believe our SuperDialseg can maintain a high quality for future research.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- A useful large-scale corpus for dialogue segmentation is proposed, with a simple but effective annotation heuristic. This would potentially benefit dialogue related tasks.\n- The paper is well-written, structured and motivated.\n- Thorough experiments across three datasets and 18 methods are conducted, providing useful findings to the community.\n", "reasons_to_reject": "Although some caveats are considered (e.g., confirmation dialogue act in Section 3.2) and human verification on a small portion (100 test dialogues) is conducted, the quality of the derived segmentation labels is bounded to that of annotations of the two DGDS corpora. The original annotation noise could be inherited and affects the segment boundary decisions.\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper presents a new dataset, SuperDialSeg, for the task of dialogue segmentation. Built upon the two document-grounding datasets, doc2dial and multidoc2dial, the authors leverage their existing annotations of the relationship between dialogue utterance and document sentence to derive the dialogue topic boundaries, with some dialogue characteristic considered and human verification on a small portion conducted. Extensive experiments are conducted on three datasets, SuperDialSeg, TIAGE, Dialseg711, with 18 different methods for comparison, including unsupervised, supervised models and LLMs.\n", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "XEo2PrWCsu", "reviewer": "Reviewer_EsRS", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your interest in our work! Our dataset, namely, SuperDialseg, and other involved preprocessed datasets (i.e. TIAGE and DialSeg711) will be publicly available immediately if our work is qualified as accepted. Our codes will also be open-sourced for this field of research. We hope our work can serve as a useful resource to develop better dialogue segmentation algorithms.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- SuperDialseg dataset and benchmarks that includes modern archtecture e.g. BERT- and GPT-based models are useful for future research. These will be made available after acceptance.\n- The human verification to check the quality of the created data is well-designed and reliable. Implementation of the verification details is also carefully explained and easy to understand.\n- The analysis is substantial (Section 5.4), and the characteristics and availability of SuperDialseg are well explained.", "reasons_to_reject": "- N/A", "questions_to_authors": "", "paper_topic_and_main_contributions": "In the dialogue segment task, it is an open issue that the definition of segmentation point is ambiguous and that there is insufficient publicly available data to train supervised learning-based models. This paper proposes a feasible definition referring to document-grounded dialogue and introduces SuperDialseg, a large dataset (approximately 1K) for supervised dialogue segment tasks based on the definitions. The novel definition of segmentation point is highly agreed upon among human annotators (Section 2.3), so it seems a practical criterion. The SuperDialseg dataset is about 10x larger than the previous dataset and of equal or higher quality (Section 3). Along with SuperDialseg, the benchmarks include 18 models across five categories of dialogue segmentation tasks are also provided. This work has the potential to contribute to the development of future dialogue segmentation research.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "S1cN0CrAzN", "reviewer": "Reviewer_mUQu", "summary": "", "questions": "Question A: could you provide a formal definition/pseudo code for your dialogue segmentation method?\n\nQuestion B: the human verification part is confusing, why not directly ask the human to accept or reject your segmentation annotation?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback! We are glad to provide further details in response to your concerns about our paper.\n\n### Question A: Could you provide a formal definition/pseudo code for your dialogue segmentation method?\n**Answer:** Yes, we are glad to do that. In Section 3.2, firstly, we introduced our motivations and the rationality of our method in Lines 236-266. Secondly, we introduced how to segment a dialogue in Lines 267-303. Thirdly, we discussed other dialogue-related information derived from the original DGDS corpora in the rest of this section. \n\nThe whole data construction method can be separated into three parts: segmenting dialogues, data cleaning, and splitting them into train/val/test subsets. In segmenting dialogues, we 1) segmented dialogues based on the grounding spans, and 2) corrected segmentation errors based on dialogue acts. In data cleaning, we removed some bad dialogues. All the stages were done automatically based on designed rules.\n\nHere is a pseudo-code for explaining how to segment a dialogue (the first part of the whole process) for your reference:\n\n```\nAlgorithm: Segment a document-grounded dialogue (DGD)\n—\nRequire: U={u_0,u_1,...,u_N}: utterances of a DGD.\nRequire: G={g_0,g_1,...,g_N}: grounding spans of each utterance, g_i is a set containing single/multiple grounding reference(s).\nRequire: Sec(g): return the section IDs of a set of grounding spans\nRequire: R={r_0, r_1,...,r_N}: roles of each utterance\nRequire: DA={da_0, da_1,...,da_N}: dialogue acts of each utterance\n—\n1:  for i=1,...,N do\n2:      if Sec(g_i) ∩ Sec(g_i-1) = ∅ do  # if they refer to different sections\n3:          L(u_i-1)=1  # A new topic begins, so the previous utterance is the end of a segment.\n4:      end if\n5:  end for\n6:  for i=0,1,...,N do   # correct segmentation errors\n7:      if da_i == 'query condition' and r_i == 'user' do\n8:          L(u_i)=0\n9:      end if\n10: end for\n11: L(u_N)=1  # The last utterance naturally locates at the end of a segment.\n—\nOutput: L: List of segmentation points.\n```\n\nWe also included the code for constructing SuperDialseg from doc2dial and MultiDoc2dial datasets in our supplementary materials. You can find the code from `supplementary/code/scripts/make_superdialseg.py`. And thanks for your comment, we will include this algorithm in our final version for future readers to understand our method more conveniently. \n\n\n### Question B: The human verification part is confusing, why not directly ask the human to accept or reject your segmentation annotation?\n**Answer:** As for the process of our human verification, we have four reasons to do that.\n1. Annotation for dialogue segmentation is not the same as that for some binary classification tasks, like sentiment analysis and response quality evaluation. Annotators need to determine the segmentation point for each utterance considering the whole conversation at the same time. Therefore, asking them to determine whether all the segmentation points are correct or not may not be applicable in dialogue segmentation.\n2. In our human verification process, we intended to obtain unbiased and natural judgments as much as possible. So we asked them to directly segment a dialogue as usual instead of asking them to accept or reject our segmentation annotation. If we provided the annotations constructed by our method in advance, it would introduce some biases (so-called Leading Bias) that may affect their judgments. For example, at first glance, they may think it ‘acceptable’/‘plausible’ and therefore exaggerate the accuracy, resulting in a misleading verification result that is not sufficiently objective and rational.\n3. Nowadays, collecting high-quality data from AMT is not as easy as before. If we only asked annotators to accept or reject our segmentation annotation, it would be hard for us to identify the spam annotations. Detailed information on how we control the quality of the human verification process can be found in Appendix D.\n4. The annotated samples from human can also be reused for future research (e.g., to understand the differences in annotations from different annotators).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Extensive benchmark experiments, which show that the supervised signals in the proposed dataset can greatly improve the model performance in dialogue segmentation.", "reasons_to_reject": "1. Unclear writing of 3.2 data construction approach, lack of formal definition of the proposed dialogue segmentation annotation method.", "questions_to_authors": "Question A: could you provide a formal definition/pseudo code for your dialogue segmentation method?\n\nQuestion B: the human verification part is confusing, why not directly ask the human to accept or reject your segmentation annotation?", "paper_topic_and_main_contributions": "This paper works on the dialogue segmentation problem.\nThe main contributions are: (1) constructing a large-scale dialogue segmentation dataset based on two existing document-grounded dialogue corpora, and (2) providing benchmark performance of 18 models in dialogue segmentation tasks.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper provides a new (supervised) dataset for dialogue segmentation.

**Pros**: All reviewers agree the dataset will be a useful resource to the community. Validation of the dataset is considered to be appropriate, using some human-verification used to verify the "simple but effective" annotation heuristic. Further, the motivations for the dataset are also well validated (the proposed supervisory signals add benefit). Indeed, reviewers note the benchmark experiments as being substantial and relevant to the community (3 benchmark datasets, 18 methods).

**Cons**: One reviewer requests missing details, while another points out that deriving supervisory signals from an existing datasets annotations (i.e., proxies) can incur noise, which will be compounded with the noise from the annotation heuristic. Authors appear to address these concerns during the rebuttal period.
Decision: Accept-Main