MultiTurnCleanup: A Benchmark for Multi-Turn Spoken Conversational Transcript Cleanup
Authors: Hua Shen
Vicky Zayats
Johann C Rocholl
Daniel David Walker
Dirk Padfield
Venue: EMNLP 2023 Main (2023)
Keywords: Multi-Turn Spoken Conversations
Crowdsourcing
Transcript Cleanup
Disfluency Detection

Abstract: Current disfluency detection models focus on individual utterances each from a single speaker. However, numerous discontinuity phenomena in spoken conversational transcripts occur across multiple turns, which can not be identified by disfluency detection models. This study addresses these phenomena by proposing an innovative Multi-Turn Cleanup task for spoken conversational transcripts and collecting a new dataset, MultiTurnCleanup. We design a data labeling schema to collect the high-quality dataset and provide extensive data analysis. Furthermore, we leverage two modeling approaches for experimental evaluation as benchmarks for future research.

MULTTTUNCCIANUP. A Benchmark for Multi-Turn Spoken Conversational Transcript Cleanup
Hua Shen
Vicky Zayats
Johann C. Rocholl* Daniel D. Walker Dirk Padfield
University of Michigan, GGoglee Research
huashen@umich ch. edu
daw.waleridwwwwwiiiiv padfieldd@@goolle.ccmm o
Abstract
B: Just in the last little while. Because know myy-ftther, uh meen, my father-in-law doesn't do that much,
Current disfluency detection models focus on individual utterances each from sin- gle speaker. However, numerous discontinu- ity phenomena in spoken conversational tran- scripts occur across multiple turns, which can not identified by disfluency detection mod- els. This study addresses these phenomena by proposing an innovative Multi-Tum Cleanup task for spoken conversational transcripts and collecting new dataset, We design a data labeling schema to collect the high-quality dataset and provide extensive data analysis. Furthermore, we leverage two model- ing approaches for experimental evaluation as benchmarks for future research.
Just in the last little while. Because I know my father-in-law doesn't that much,
(a) Disfluency Detection Task
B: think that's changed just in the last generation.
A: Exaetly
B: dteat-in-the-lsse wetess----t--hhllleee Because I know may-father; mhe-m my father-in-law doean't that much,
A: Exeetly
1 Introduction
I think that's changed just the last generation.Becc Because know my father-in-law doesn't do that much,
Spontaneous spoken conversations contain in- terruptions such filled pauses, self-repairs, etc. (Shriberg, 1994). These phenomena act noise that hampers human readability (Adda- Decker et al., 2003) and the performance down- stream tasks such as question answering (Gupta et al., 2021) or machine translation (Hassan et al., 2014) on transcripts of human spoken conversa- tions. State-of-thr -ftie-of-hhearr art disfluency detection meth- ods (Yang et al., 2020; Jamshid Lou and Johnson, 2020) identify and remove disfluencies in order to improve the readability of spoken conversational transcripts (Wang et al., 2022; Chen et al., 2022). For instance, Figure ((a) shows that disfluency de- tection methods can remove self-repairs of single turns. However, these models focus on remov- ing interruptions and errors that commonly occur within single-turn utterances and cannot handle dis- continuities across multiple turns. For example, in Figure 1(b), speaker B is in the middle of thought when Speaker A interrupts to signal that they are
(b) Multi-Tum Cleanup Task
A comparison of (a) existing Distluency De- tection Task (yellow highlights indicate disfluencies) with (b) the proposed Multi-Turn Cleanup task (red highlights indicate multi-turn cleanups) for spoken con- versational transcripts.
Figure 1:
following along ("A: Exactly"). B continues their train of thought ("B: Just in the last little while. Be- cause...") by paraphrasing their own last sentence ('...just in the last generation."). The result an exchange that is longer and more difficult to follow than necessary to understand what B is conveying.
This paper aims to "clean up" spoken conversa- tion transcripts by detecting these types of multi- turn "discontinuities" inherent in spontancous spo- ken conversations. Once detected, they can be moved to produce transcripts that look more like hand-written conversations conducted over text messaging, social media, or e-mail as shown Figure 1(b). Given that this is a novel task, with no
This work was done when the first author was a research intern at Google Research.
Wee relcase the collected MuETTURNCIAAN dataset at: https://github..o htwp:thhwww.ooo/uusshhheaaa 18M.aeennn22//M//iiaaeei (umMMlullTuumCCauuu.....

---

Compared with the existing disfluency detection task, which aims to detect disfluencies (e.g., self- repairs, repetitions, restarts, and filled pauses) that commonly occur within single-turn utterances (Ro- choll et al., 2021; Chen et al., 2022), the Multi- Turn Cleanup task requires identifying disconti- nuities both within a single turn and across multi- ple turns in the multi-party spoken conversational Count (%)
Category
Definition
Conversation Instance
Speakers show that they listening to and agree with the other speakers
24.3k (17%)
A: guess both of us very much aware of the equality. it seems like women arc, just starting get kind of equality jobs and the home where husbands are starting doing dishes, er-seme
Acknowledgment
and Confirmation
30k (21%)
B: think that's changed just the last generation.
Repetition and Paruphrase
Speakers may repeat or para- phrase their words during the conversation.
A: Exactly-
B: Just in the last little while. Because
I know my father-in-lave doesn't 't do that much,
15.7k (11%)
Think aloud
Speakers talk to themselves during thinking instead of talking to others.
A: Exactly-
B: of dishes, taking care of kids, or-what clse, you knew, that kind of stuff but my husband is wonderful.
47.2k (33%)
Incomplete
Speakers may also say incom- plcte sentences due to inter- ruption, changing topics, etc.
A: that's 's the way my hushand is tno. it doesn't bother him to do the dishes, it-dess bother him the laundry verses, men from way back there is that, if you did that you were henpecked.
Sentences
The remaining discontinuity categories.
25. (18%)
Others
Table 1: The linguistic taxonomy (Category) and definition (Definition) of discontinuities in the MULITTUR- CleAnUP dataset for the Multi-Turn Cleanup task. We further provide the statistics of each category (Count( %)) in the dataset and conversational instance (Conversation Instance), where A and B indicate two speakers.
pre-existing labeled data or benchmarks, we first define taxonomy of non-disfluency discontinu- ities (see Figure 1). Then we collect new dataset, MULnTURNCLEANUR, for the Multi Turn spoken conversational transcript Cleanup task, based on the Switchboard Corpus (Godfrey et al., 1992) and label according to the proposed taxonomy. Finally we develop two baseline models to detect these discontinuities which we evaluate as benchmarks for future Multi-Turn Cleanup task studies. Our data analysis suggests that the MULnTURACLEANUU dataset is of high quality. We believe it will help to facilitate research in this under-investigated area.
transcripts. To explicitly define the task and dis- continuity taxonomy, we conducted an in-depth analysis of the Switehboard corpus (Godfrey al., 1992). Specifically, we randomly sampled a sub- set of Switchboard conversations, annotated the discontinuity spans other than existing disfluency types, and grouped the annotated discontinuities into five main categories. Note that we conducted the discontinuity annotation and category grouping process iteratively with all authors to reach the con- sensus. We demonstrate the finalized taxonomy of discontinuities in Table 1.
2.2 Data Preprocessing
2 Data Collection and Analysis
We preprocessed the Switchboard corpus by au- tomatically removing the single-turn disfluen- cies with pre-defined rules based Treebank- 3 (Mitchell et al., 1999)4. As result, could encourage our recruited humans annotators con- centrate on cleaning up multi-turn discontinuity phenomena based on the five categories in Table 1.
We propose an innovative Multi-Tur Turn Cleanup task and collect a novel dataset for this task called Mun.- nTTURNCLEANUP". This section presents the task definition, data collection process, and analysis.
2.1 Task Definition
We split each conversation into multiple chunks, where each chunk composes one HIT (Human In- telligence Task) containing around 300 tokens. We further ensure that the successive chunks overlap around 50% of tokens, providing enough context for each conversation fragment. The resulting data
wSitchboard corpus a collection five-minute human- human telephone conversations. We chose Switchboard for data construction because: 1) it has relatively large size; and 2) contains ground -ruuh disfluency annotations that our study can build upon to annotate multi-turn cleanup labels.
"The data collection cost was about $18, 000, with pay- ment as 40,41/HIT (S8/hour assuming 3min/HIT) and honuses ($2 - $100/worker) for top raters.
"See Appendix A,2 for detailed preprocessing rules and data statisties.

---

Data and Interface Preparation
Qualification Selection
Batch-wise Labeling with Quality Checkpoint Filter
Pilot Studies
Done
Repost
Version 1
Testing with different versions
Design difterant Ulidaa processing
Start Ckipt
Middle
End
Filter warkers with espert-iabeled ground fruth annotation.
Version M
Quality Checks
DatafUI Revisions
Figure 2: A four-step data labeling schema collect MULTTORNCLEANUU dataset for the Multi-Tum Cleanup task on spoken conversational transcripts. This schema enabled efficient collection of the high-quality MULTiTUR- CLEANUP dataset via MTurk platform, where annotators labeled cleanup marks and corresponding categories.
Datasets
#Conv
#Turns
#Tokens
#Cleanup
IRR
Experts
MTurk Workers
Train
932
74k
IM
132k
Train
Dev
Test
Fleiss'
Dev
86
3.7k
60k
6.1k
0.596
Kappa
0.560
0.592
0.557
0.561
Test
64
2.9k
43k
5k
Sum
1082
81k
1.1M
143k
Table 3: The averaged Fleiss' Kappa scores of all con- versation turns in the MULIITURNCLEANUP dataset.
Table 2: Statistics of MULITURNNLEANU dataset.
preprocessing statistics are shown in Table 6 in Appendix A.2.
batches including 7277 HITs and 11 QCH in tal. We leverage these checkpoint HITs exclude unqualified workers (FI < 0.3).
2.3 Labeling Procedure
Annotation filtering and aggregation. After fin- ishing the final batch, we collected all annotated batches and excluded 72 unqualified workers with all their HITs. Then we reposted 26% of the signments where the conversations had less than two annotations the remaining qualified work- ers. Finally, we aggregated the annotations for each turn by only keeping the best worker's (highest F1- score) labels to compose the dataset. The average F1 for raters of labeled turns in MULITTURXCLENUU is 0.57. We summarize the per-category statisties in Table I and the statistics of MULITTRNNIEANC in Table 27.
Given the preprocessed data, we then conducted the human annotation process based on a data labeling schema shown in Figure 2.
Preparation and qualification selection. In steps and 2, prepared suite of data preprocessing and user interface (UT) variations and conducted seven pilot studies to select the optimal task de- sign. The final UT (see Appendix A.4) consists of: i) an introduction to the task, ii) an annotation example with highlighted discontinuities, and iii) the task workspace with affordances for annotation. In step 3, we recruited set of qualified MTurk workers using "Qualification HIT". We compared all 580 workers' submissions with the ground truth (authors' consensus) and select the 222 workers (38.3%) with an Fl 0.3" to participate in step 4. Large-scale data labeling. Controlling annota- tion quality for large-scale data labeling is chal- lenging in MTurk (Daniel et al., 2018). ad- dress this, we employed a "batch-wise labeling with quality checkpoint filter" (Bragg and Weld, 2016). Specifically, we split the dataset into small batches and posted them with "Quality Checkpoint HITs" (QCH) mixed in. Overall, we posted 22
2.4 Validating Human Annotation Accuracy
During the whole data labeling process, sistently assessed the human annotation accuracy and filtered out unqualified workers control the data quality. We visualize the annotation quality in terms of the distribution of workers' FI scores (see Figure (lA)tt) as well as the correlation between each worker's F1 score and their finished HIT counts and the average elapsed time per HIT (see Figure ((A)(B)-right). These figures show how removing unqualified annotations with checkpoints can effectively control quality during the annota-
*Steps and 4 lasted about one month. More annotation quality control details are available in Appendix A.3. "The 0.3 threshold is reasonable due to task subjectivity.
We leave out sw4[2-4J subgroups in Switchboard as they are loss commonly used, resulting in 1082 total conversations.

---

Worker Quality Levels
Correiation Analyss
Worker Quality Levels
Correlation Analysis
Fount ol Duathet Wecoors
Worder 111 Courts
Worter HIT Counts
Oualifed
00.3.40 1.21
S.Scam Threshol
F1 sc.ccoe Tireshold
FT Thaeshood
(A) Start Quality Checkpoint
(B) End Quality Checkpoint
Figure 3: The comparison of worker quality performance between (A) start checkpoint and (B) end checkpoints. For each one, we plot participated workers' Fl score distribution (left) and the correlation between each worker's s score and finished HIT count (right), the circle size means each worker's averaged elapsed time to finish a HIT.
Sub-tasks
Model
F1
R
P
Model
F1
R
P
DISPLUENCY
STD
89.8
88.3
91.2
Multi-Turm
BASELINE
58.2
42.5
92.3
Cleanup
MULTTORN
BASELINE
15.5
8.77
65.8
Two -ta-Sttge
64.6
72.3
Task
CLEANUP
MTD
56.8
55.4
58.3
Combined 74.9
72.9
76,9
Table 4: Model performance on the two sub-tasks, in- cluding detecting single-turn disfluencies with DISFLU- ENCY dataset and multi-turn discontinuities with the pro- posed MULNTURNCI FANUP dataset.
Table 5: Modle performance on the overall Multi-Tum Cleanup task with the UNIeNDIScONTNUTTY dataset, where the Combined Model achieves the best F1 score.
BERT-based modeling, presented in Rocholl et al. (2021), for both STD and MTD stages. Partic- ularly, fine-tune the STD based on the tradi- tional single-tum disfluency dataset (Godfrey et al., 1992), whereas the MTD is fine-tuned based on our collected MULITTUnNCIFANU dataset. We concate- nate STD and MTD successively into the pipeline of the two-stage model, so that both the single-turn disfluencies and multi-turn discontinuities in the raw conversational transcript can be removed with one pass.
tion process. Particularly, we observe that at the start (A), even after passed our initial "Qualifica- tion HIT in step 3. 23% of workers perform at Fl < 0.3 but complete over 80% of all assignments, leaving only limited amount of data for more com- petent workers to label. By continually excluding unqualified workers with
0.3, all remaining workers have F1 2 0.3 by the final batch (B).
2.5 Turn-based Inter-Rate Rater Reliability
We compute Inter-Rater Reliability using Fleiss' Kappa Agreement (Fleiss and Cohen, 1973) for each annotated turn and average all turns' scores. Table 3 shows that the workers' Fleiss' Kappa scores comparable to those of the authors.
3.2 The Combined Model
We design the combined model, using only one BERT-based detector (Rocholl et al., 2021), to si- multancously remove both single-turn disfluencies and multi-turn discontinuities. To this end, we create UNIoNDISCONTINUII dataset, which com- bines both the single-turn disfluency and multi-tum discontinuities labels in Godfrey et al. (1992) and MULn TURNCLENUU datasets, respectively. Then we achieve the combined model by fine-tuning the detector with this UNIoNDiscONTINUTyY dataset.
3 Multi-turn Cleanup Models
Given the collected CuleTurnCleanuP dataset, we leverage two different BERT-based modeling approaches, including two-stage model and combined model, for the Multi-Ture Cleanup task to remove both single-turn disfluencies and multi- turn discontinuities.
4 Experiments
3.1 The Two-Stage Model
4.1 Experimental Setup
The two-stage model is composed of Single-Turn Detector (STD) to remove the traditional single- turn disfluencies and successive Multi-Turn De- tector (MTD) to remove the discontinuities oc- curring across multiple turns. We employ the
The Two-Stage Model. The STD and MTD are separately trained. We train the STD with the ex- isting disfluency dataset, where the input is sin- gle sentence (i.e. slash unit), with a maximum sequence length of 64, In comparison, we train the

---

MTD with MULITTURNCLENNUP dataset, where the input consists of multiple slash-units (demarcated with [SEP] token between turns) with maximum sequence length of 512. We feed full transcripts to the MTD in chunks with an overlap of 50% for prediction context. Then we predict discontinuities where either of the overlapping predictions for given token was positive. During inference, the stage-2 MTD module loads the outputs from the stage-1 STD module, removes all of the tokens classified as disfluencies, and uses this redacted texts as its own input.
dataset. The significant disparity between MTD and BASELINE methods (e.g., 56.8 vs. 15.5 in FI) also indicate the difficulty of detecting multi-turn discontinuities in MULITTUNNCLAANC dataset. Evaluation on removing all discontinuities. Fur- thermore, we evaluate the overall model perfor- mance on jointly detecting the single-turn disfluen- cies and multi-turn discontinuities with one pass based the UNTONDISCONTINUTIY dataset. As shown in Table 5, we observe that both the pro- posed Two-Stage Model and Combined Model can outperform BASELINE method. In addition, the Com- bined Model achieves a 6.7 higher FI score than Two-Stage Model on the Multi-Turn Cleanup task.
The Combined Model. We train the combined model with the UNHONDISCOTINUTTY dataset us- ing the same training settings of the aforemen- tioned MTD module. During inference, we predict both single-tur turn and multi-turn discontinuities, as nondistinctive labels, simultancously.
5 Related Work
Recent disfluency detection studies develop BERT- based models (Bach and Huang, 2019; Rocholl et al., 2021; Rohanian and Hough, 2021) and show significant improvement over LSTM-basce mod- els (Zayats et al., 2016; Wang al., 2016; Hough and Schlangen, 2017) in disfluency detection tasks, Prior studies also show the importance of data augmentation methods, by leveraging extra tran- script sources to improve disfluency detection per- formance (Jamshid Lou and Johnson, 2017, 2020), While most of the research has been focused on improving single-turn disfluency detection accu- racy, little exploration has been done in detecting multi-turn transcript discontinuities.
Baseline. We employ the state-of-the-art BERT based disfluency detection model (Rocholl et al., 2021) trained with the widely used disfluency dataset (Godfrey et al., 1992) as the BASELINE. Deployment. We train the models on Google's s Au- toML platform, where it selects the optimal training settings as: Adam optimizer with learning rate as le 5, batch size of 8, and epoch.
4.2 Evaluation Metrics
We evaluate all models' performance with per- token Precision (P), Recall (R), and F1 score (F1) on predicting if the token should be cleaned as single-turn disfluencies (STD of two-stuge model), or multi-turn discontinuities (MTD of two-stage model), or their mixtures (the combined model).
Obtaining reliable annotated datasets via crowd- sourcing is challenging and expensive (Alonso et al., 2014; Wong et al., 2022; Northcutt et al., 2021). To collect qualified dataset for multi-turn cleanup task, this work designs a data labeling schema which efficiently collects qualified dataset via the MTurk.
4.3 Results
Evaluation on two sub-tasks. The Multi-Turn Cleanup task inherently involves two different sub- tasks, including the single-tum disfluency detection (i.e., with DISFLUENCY dataset) and multi-turn dis- continuity detection (i.e., with our collected Mul- TTTURNCLEANUP dataset), we first validate that the presented models can achieve state-of-the-an per- formance on the two sub-tasks (i.e., with the two different datasets), respectively.
Limitation and Conclusion
We are aware that, in some specific scenarios, it might be undesirable to remove some multi-turn discontinuities because they convey social mean- ing in human interactions (e.g., engagement). We address this issue by providing category labels. As a result, future research can flexibly select subsets of the MunnttunClLANN labels to train the model and clean up multi-turn discontinuities.
Particularly, Table 4 illustrates the performance of BASELINE and presented models the two datasets. The STD module achieves cutting-edge performance (Chen et al., 2022) detect single- turn disfluencies. Also, the MTD module outper- forms the BASELINE on detecting multi-turn dis- continuities with our proposed MulTTURnCLEANUP
This study defines an innovative Multi-Turn Cleanup task and collects a high-quality dataset for this task, named MULTTTRNCIEANC using our pre- sented data labeling schema. We further leverage two modeling approaches for experimental evalua- tion as the benchmarks for future research.

---

Acknowledgements
North American Chapter the Association for Com- putational Linguisties: Human Language Technolo- gies, pages 827-838, Seattle, United States. Associ- ation for Computational Linguistics.
We thank Noah B. Murad for his help in conducting the human evaluation experiments. We also thank Shyam Upadhyay, Daniel J. Liebling, Tiffany Kn- earem, Kenton Lee, and other Google colleagues for providing their constructive feedback on this study, We thank the Amazon MTTurk workers for their excellent annotations. We thank the reviewers for their thoughtful comments.
Florian Daniel, Pavel Kucherbaev, Cinzia Cappiello, Boualem Benatallah, and Mohammad Allahbakhsh. 2018. Quality control in crowdsourcing: A survey of quality attributes, assessment techniques, and as- surance actions. ACM Computing Surveys (CSUR). 51(1):1-40.
Joseph I. Fleiss and Jacob Cohen, 1973. The equiv- alence of weighted kappa and the intraclass corre- lation coefficient measures of reliability. Educa- tional and psychological measurement, 33(3)6113- 619.
Ethics Statement
The collected MULITTURNCLEANUP dataset is built upon the published Switchboard Corpus (God- frey et al., 1992). The dataset is sufficiently anonymized, so it is impossible identify indi- viduals. In addition, we protect privacy during the data collection process through the MTurk platform. The posted dataset does not list any identifying in- formation of MTurr workers. Also, the data collec- tion process does not access any demographic or confidential information (e.g., identification, gen- der, race, etc.) from the MTurk workers. In general, the dataset can be safely used with low risk in re- search and application fields for cleaning up spoken conversations and speech transcripts.
John J Godfrey, Edward C Holliman, and Jane Mc- Daniel. 1992. Switchboard: Telephone speech cor- pus for research and development. In Acoustics, Speech, and Signal Processing, IEEE International Conference on, volume pages 517-520. IEEE Computer Society.
Aditya Gupta, Jiacheng Xu, Shyam Upadhyay, Diyi Yang, and Manaal Faruqui. 2021. Disfl-QA: A benchmark dataset for understanding disfluencies in question answering. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 3309-3319, Online, Association for Computa- tional Linguistics.
Hany Hassan, Lee Schwartz, Dilek Hakkani-Túir and Gokhan Tur. 2014. Segmentation and disfluency removal for conversational speech translation. In Fifteenth Annual Conference of the International Speech Communication Association.
References
Martine Adda-Decker, Benoft Habert, Claude Barras, Gilles Adda, Philippe Boula de Mareuil, and Patrick Parouhek. 2003. A disfluency study for cleaning spontaneous speech automatic transcripts and im- proving speech language models. In ISCA Tutorial and Research Workshop on Disfluency in Sponta- neous Speech.
Julian Hough and David Schlangen. 2017. Joint, incre- mental disfluency detection and utterance segmenta- tion from speech. In Proceedings the 15th Confer- ence of the Earopean Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 326-336.
Omar Alonso, Catherine Marshall, and Marc Najork. 2014. Crowdsourcing subjective labeling task: a human-centered framework to ensure reliable re- sults. Microsoft Res., Redmond, WA, USA, Tech. Rep. MSR-TR-2014-91,
Paria Jamshid Lou and Mark Johnson. 2017. Disflu- ency detection using noisy channel model and deep neural language model. In Proceedings of the 55th Annual Meeting of the Association for Compu- tational Linguistics: ACI. 2017, (Volume 2: Short Papers), pages 547-553, Vancouver, Canada. Asso- ciation for Computational Linguistics.
Nguyen Bach and Fei Huang. 2019. Noisy bilstm- based models for disfluency detection. In INTER- SPEECH, pages
Paria Jamshid Lou and Mark Johnson. 2020. Im- proving disfluency detection by self-training self- attentive model. In Proceedings of the 58th Annual Meeting of the Association for Computational Lin- guistics, pages 3754-3763, Online. Association for Computational Linguistics.
Jonathan Bragg and Daniel S Weld. 2016. Optimal test- ing for crowd workers. In Proceedings of the 2016 International Conference on Autonomous Agents Multiagent Systems, pages
Angelica Chen, Vicky Zayats, Daniel Walker, and Dirk Padfield. 2022. Teaching BERT to wait: Balancing accuracy and latency for streaming disfluency detec- tion. In Proceedings of the 2022 Conference of the
Marcus Mitchell, Beatrice Santorini, M Marcinkiewicz, and Ann Taylor. 1999. Treebank-3 Idc99142 web download. Philidelphia: Linguistic Data Consor- lium, 3:2.

---

Curtis Northcutt, Anish Athalye, and Jonas Mueller. 2021. Pervasive label errors test sets destabilize machine learning benchmarks. In Thitty-fifth Con- ference on Neural Information Processing Systems Datasets and Benchmarks Track (Round) 1).
A
Appendix
A.1
Definitions of Disfluency Detection and Annotations
Disfluency Definition. When humans speak, our language peppered with interruptions and errors known as disfluencies. Formally, disfluencies are irregularities that are an integral part of sponta- neous speech and include self-repairs, repetitions, restarts, and filled pauses (Schegloff et al., 1977).
Johann C. Rocholl, Vicky Zayats, Daniel D. Walker, Noah B. Murad, Aaron Schneider, and Daniel J. Liebling. 2021. Disfluency Detection with Unla- beled Data and Small BERT Models. Proc. In- terspeech 2021, pages 766-770.
Morteza Rohamian and Julian Hough. 2021. Best of both worlds: Making high accuracy con-incremental wranformer-hasel disfluency detection incremental. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and 1 1ih International Joint Conference on Natural Lan- guage Processing (Volume 1: Long Papers), pages 3693-3703.
Disfluency Annotations. Following Shriberg et al. (1997), the disfluency annotation includes:
the reparandum: the material that the speaker intends to delete.
the interruption point: denoted as (+).
Emanuel Schegloff, Gail Jefferson, and Harvey Sacks. 1977. The preference for self-correction in the organization of repair in conversation. Lan- guage, 53(2)3661-382.
optional interregnum: enclosed in curly brackets, which include filled pauses and dis- course markers, such as "uh", "um", "you know", "I mean", etc.
Elizabeth Shriberg, Rebecca Bates, and Andreas Stol- cke. 1997. A prosody only decision-tree model for disfluency detection. In Fifth European Conference on Speech Communication and Technology.
optional the material that semantically replaces the reparandum.
Elizabeth Ellen Shriberg. 1994, Preliminaries to a the- ory of speech disfluencies. Ph.D. thesis, Citeseer.
Some examples of disfluency annotation include:
Shaolei Wang, Wanxiang Che, and Ting Liu. 2016. neural attention model for disfluency detection. Proceedings of COLING 2016, the 26th Inter- national Conference on Computational Linguistics: Technical Papers, pages 278-287.
[ lits + { uh } it's almost
[ was it, + { I mean, 3 did you I put...
just enjoy working
By + it was attached to
Zhongyuan Wang, Yixuan Wang, Shaolei Wang, and Wanxiang Che. 2022. Adaptive unsupervised self- training for disfluency detection. In Proceedings of the 29th International Conference on Computa- tional Linguistics, pages 7209-7218, Gycongju, Re- public of Korea. International Committee on Com- putational Linguistics.
A.2
Dataset Preprocessing Details.
Datasets
#Conv
#HIT's
#Turns
#Tokens
Train
932
6579
142.135
1,928,169
Dev
86
7,250
115,756
Ka Wong, Praveen Paritosh, and Kurt Bollacker. 2022, Are ground truth labels reproducible? an empiri- cal study. In MI. Evaluation Standards (ICLR 2022 Workshop).
Test
64
302
5,888
85,321
Sum
1082
7277
155,273
2,129,246
Table Statistics data preprocessing results.
Jingfeng Yang. Diyi Yang, and Zhaoran Ma. 2020. Planning and generating natural and diverse disflu- ent texts augmentation for disfluency detection. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1450-1460.
We preprocess the Switchboard corpus (Godfrey et al., 1992) with the human-annotated disfluencies based on Treebank-3 (Mitchell et al., 1999).
Based on the human annotations, we first cleaned the sentences by removing the non-speech events and words/phrases with the markers including:
Vicky Zayats, Mari Ostendorf, and Hannanch Ha- jishirzi. 2016. Disfluency detection using a bidirec- tional Isum. Interspeech 2016, pages 2523-2527.
Prosodic markup, like # and /.
Other language markup. like <English bike>.

---

Noise
markers,
like <laughter or
A.4 User Interface Demo
Throat clearing>
We demonstrate the finalized User Interface (UI)
Noise markers with curly brackets, like [breathing] or (gasp) or (lipsmack) or [again, imitates the sound of whales)
design of the HIT for the large-scale datasets Figure 4.
Double parentheses from uncertainty markers, like ((yesterday))
Plus signs surrounding words, like +ighht- seeing+.
Context markers, like (laughter) or (RE- CESS).
Punctuations, like %, w*, &,,>, +>, <1>, O. (G)), [[.
In the next stage, we conservatively remove the following types of disfluencies recursively:
Reparandum the material that the speakers intends to delete.
Interregnum including types of:
Discourse markers (i.e., marked with
(D...) like you know, well, so, like, etc.).
Explicit editing term (i.e., marked with (E...) like mean, sorry, excuse me, etc).
- Filler words (i.e., marked with (F...) like uh, um, huh, oh, etc.
By removing all the markers and disfluencies de- scribed above, we present the remaining contents Switchboard corpus to MTTurk workers for label- ing the MULNTTURNCLANUU dataset for Multi-Turn Cleanup task.
A.3
Key notes of data labeling.
We notice a list key points that are imperative to ensure the data quality and avoid spammers in MTur platform.
Keep task simple and instructions clear. This helps workers better understand the task, reduce their cognitive load and focus on anno- tating qualified outputs.
Select workers and check quality constantly. Repeatedly inspecting worker quality can sig- nificantly reduce spammers and improve data quality, more details validated in See 2.4.
Communicate via emails. Notifying new post or notes workers helps improve return rate significantly. Also, their feedback may be valuable to improve the data labeling.

---

Instructions
Conversation Transcript Cleanup
this HIT, you will med one instructoon followed by Biree sudtaska For each subtask you are asked to "clean up conversation transcripe anippet.
The speakeers A and right stop change thoughts during the conversation, repest words, or correct something they atready uand. etco Your taok is CROSS OUT all the unnocessary phrases without affecting the overall sext meaning As result the dleaned conversations would be more cohorent conciso,
movie scripts podicasts.
Meanwhile, you will also annotate which categary of deanup you think phrase is by clicking the category button first and than cresa the typical categoriss
We woskzate annotated transcripts decide the qualification based annotations accurate and otherss sategories
questions feadbacks,
dichtencysaoliinnnnnnggggg Thank
Transcript Cleanup Examples Exampless
Conversation Example
going police,
Shates thim you'te
poing
getiing tngether trving asking hug
It's going to take all of us, getting together and just sayling, you imore we're not going take more.
allow neighborhood
thet' thete
and they'r going other neighborhoode
what' neightbborhodd names from LakeVieww neighborhood,
and then quite often theyll shat, because they're horning sombody aisels temiitor..
Cleanup Types and Definition
Acknow lednowlelgeeeeett and Confirmation
words/ fsentences show Estening and agrse speakiars, fihat's thin youree) thou ledding!, MMl gedi) atc.
adinowlldgemee
2. Rlepettion and Paraphrase
conversation. "from Laketiew neightorhood".
repeoted lits
atrof getting tngether! Also, repetition
3. Think aloud
Think aloud includes words/ sentencess that the speakers themssef duaing their thinking but not talking other speaker for say
Innompiett Sentences
Speakers may also say incomplate santences due to innemuption or changing topica, etc. For example, you should remove the incomplete sentence fim speaker
categories cleanupo conversation.
type. example, fyou know!
add meaning
Examples
Please select the cleanup
Acknnwweedgement
Repetitin
Think Aloud
Incompliteness
Others
Edit transcript
(this window is draggable)
Okay. Did you want go ahead?
ahead start
Okay, one thing. guees both are very much aware equality. seeme women are, just starting to really get some of equality not only In, jobs but in the home where husbands are starting help out more than they over did. it's not uncommon see men doing dishos, makinng supper, or taking care of kids or som. the genoration my parents and my great grandmother they did everything. They went to work and they took care of the family, so.
changed
Exactly
my
Exactly.
my
that's the husband too. doesstt bother him at all the distes. It doesnt bother him all do the laundry verses, men from way book. There that, that you hanpecked whatever But, men starting.
that cooperating
those very few. think those don't lang relationahips because I think women know what they want.
They're accepting that anymore.
esactly, hard enough
everything, and kind light There's that households,
kind
better.
You submit seconds
Figure 4: An example of the User Interface for the MTurk worker annotation. The "Review Transcript Cleanup Example" section is foldable by clicking to hide and show the example.


Reviews:
[{"review_id": "p4uiizRkHz", "reviewer": "Reviewer_chnj", "summary": "", "questions": "The work is conducted on the Switchboard corpus. Is there a reason, the ICSI corpus has not been used? In the past, it has been studied extensively for disfluencies. Similar to the TRAINS corpus, where also a range of scientific papers have been published on this topic. \n\nWhat is the goal of removing disfluencies in such a way? Is it to aid summarization? To improve some other down-stream tasks? \n\nWhy was the data annotated with the specific annotation scheme? A lot of the phenomena covered in the annotations, could also be handled using Dialog Acts and more specifically, Backchannels, which some of these phenomena are.  \n\nThe paper reports the IAA based on Fleiss Kappa for the annotation. From the weighted average it is unclear which category achieved which agreement and what the distribution of the various categories is. It would be enlightening, if some more statistics were shared. \n\nAre there plans to publish the research artifacts linked to this work?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their valuable questions and feedback. Please see our response per question/concern below.\n\nReviewer: \"It is unclear if the data cleaned with the proposed method is actually easier/better to process for down-stream tasks…\"\n\nResponse: Our main motivation in this work in the first place was to aid human comprehension of transcribed speech by cleaning up unnecessary parts of transcripts that do not convey semantic meaning or are redundant, making the transcript more readable by humans. While it is likely that such a clean-up technique would also benefit the downstream spoken language understanding tasks (e.g. summarization), we currently limit the scope of such explorations in this short paper and leave it for future work.\n\nReviewer: \"Is there a reason the ICSI corpus has not been used? In the past, it has been studied extensively for disfluencies. Similar to the TRAINS corpus, where also a range of scientific papers have been published on this topic.\"\n\nResponse: Switchboard is one of the widely-used datasets for disfluency detection and has been used extensively in prior work (see for example [1,2,3] below). Switchboard consists of spontaneous multi-party speech over the phone between strangers over a diverse set of topics. While ICSI corpus also consists of multi-party conversations, the dataset is very domain-specific (a lab's group meeting recordings), thus making it a less ideal choice for the only corpora being annotated with multi-turn discontinuities. The same concerns apply to the TRAINS set. We hope that our initial annotation schema developed and used in Switchboard corpus is general enough and can be applied in future work to annotating other datasets such as the ones mentioned by the reviewer.\n\nReviewer: \"A lot of the phenomena covered in the annotations, could also be handled using Dialog Acts and more specifically, Backchannels, which some of these phenomena are.\"\n\nResponse: We indeed considered automatic data preprocessing approaches to handle some multi-turn phenomena (see details in Appendix A.2). Although the backchannels can handle part of the “Acknowledgment” discontinuity type categorized in Table 1, we found that the remaining types cannot be resolved by Dialogue Acts, such as “Think Aloud” and Cross-turn “Repetitions” (an example in figure 1B). Therefore, we recruit humans and guide them to focus on removing these more complicated multi-turn discontinuities.\n\nReviewer: \"The paper reports the IAA based on Fleiss Kappa for the annotation. From the weighted average it is unclear which category achieved which agreement and what the distribution of the various categories is…\"\n\nResponse: We compute the IRR score on Fleiss Kappa in order to show that the annotation quality resulting from our schema is comparable to experts. We will add more reviewer suggested statistics given more space in the extra page upon acceptance.\n\nReviewer: \"Are there plans to publish the research artifacts linked to this work?\"\n\nResponse: We have released the annotated dataset described in this paper.\n\nReviewer: \"While the authors state how much the annotators (turkers) were paid, it is open, whether they were paid adequately, if they were treated fairly etc.\"\n\nResponse: The hourly wage paid to the workers is comparable with the average wages usually used in industry for similar types of annotation work. Figure 4 presents the exact prompt presented to the anonymous workers who completed the annotation through the Amazon Mechanical Turk platform. We are not sure what the reviewer means by \"if they were treated fairly\" since all the infrastructure and employment relationship is handled by the Amazon Mechanical Turk platform, while the authors only provide the task (Figure 4) and the price per task ($0.40 per task + bonuses).\n\nReferences:\n\n[1] Honnibal, Matthew, and Mark Johnson. \"Joint incremental disfluency detection and dependency parsing.\" Transactions of the Association for Computational Linguistics 2 (2014): 131-142.\n\n[2] Zayats, Vicky, Mari Ostendorf, and Hannaneh Hajishirzi. \"Disfluency detection using a bidirectional LSTM.\" Interspeech (2014).\n\n[3] Dong, Q., Wang, F., Yang, Z., Chen, W., Xu, S., & Xu, B. (2019, July). Adapting translation models for transcript disfluency detection. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 33, No. 01, pp. 6351-6358).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "With the rise of more and more spoken material and spoken interactions dealing with peculiarities of spoken language is an important task. The paper presents a dataset, annotations and baseline experiments to deal with discfluencies, a rather prominent phenomenon in spoken language, which disturbs most NLP pipelines. ", "reasons_to_reject": "It is unclear if the data cleaned with the proposed method is actually easier/better to process for down-stream tasks, as the paper only presents an intrinsic evaluation, but not an extrinsic. As such, it is unclear to what end the disfluency removal is being carried out. ", "questions_to_authors": "The work is conducted on the Switchboard corpus. Is there a reason, the ICSI corpus has not been used? In the past, it has been studied extensively for disfluencies. Similar to the TRAINS corpus, where also a range of scientific papers have been published on this topic. \n\nWhat is the goal of removing disfluencies in such a way? Is it to aid summarization? To improve some other down-stream tasks? \n\nWhy was the data annotated with the specific annotation scheme? A lot of the phenomena covered in the annotations, could also be handled using Dialog Acts and more specifically, Backchannels, which some of these phenomena are.  \n\nThe paper reports the IAA based on Fleiss Kappa for the annotation. From the weighted average it is unclear which category achieved which agreement and what the distribution of the various categories is. It would be enlightening, if some more statistics were shared. \n\nAre there plans to publish the research artifacts linked to this work?", "paper_topic_and_main_contributions": "The paper \"MultiTurnCleanup: A Benchmark for Multi-Turn Spoken Conversational Transcript Cleanup\" presents a workflow to tackle disfluencies not only in single utterances/utterances by the same speaker, but also disfluencies spread across multiple utterances/several speakers. To that end, the paper presents an annotation scheme, manual annotations and an automatic detection model for finding and removing all disfluencies. The experiments are based on the Switchboard corpus, which has been labelled according to the presented annotation scheme. \n", "excitement": 3, "reproducibility": 3, "ethical_concerns": "Yes", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "References to work on Disfluencies related to the ISCI and TRAINS corpus, i.e. https://aclanthology.org/N13-1083.pdf might be a good starting point. ", "Justification_for_Ethical_Concerns": "While the authors state how much the annotators (turkers) were paid, it is open, whether they were paid adequately, if they were treated fairly etc. "}, "extra_text": {}}, {"review_id": "GyyFFreS9Q", "reviewer": "Reviewer_H2MH", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for your insightful comments and suggestions. Although audio data is beyond this short paper’s scope, adding non-phonemic information like prosody to improve disfluency detection performance seems to be a promising direction, and we would like to plan this exploration in our future work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Introduction of new dataset that will facilitate future research in multi-turn discontinuity detection, an important problem in my opinion.\n2. Very detailed description of their data annotation process that can benefit future data collection experiments.", "reasons_to_reject": "1. It would have been interesting if they had also included audios in their dataset since some recent work has shown that non-phonemic information like prosody can improve disfluency detection performance.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper proposes to clean up spoken conversation transcripts by detecting multi-turn discontinuities found in spontaneous spoken conversations.\n\nThe authors first define a taxonomy of multi-turn discontinuities and then collect a dataset based on this taxonomy.\n\nThe authors then train 2 baseline models using this dataset and find that the dataset can be used to facilitate future research in building multi-turn disfluency detection systems.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "OMbS5LsKvd", "reviewer": "Reviewer_Hvuk", "summary": "", "questions": "na", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for your constructive feedback and suggestions. \n\nWe only leveraged Bert-based models mainly due to two reasons: \n\n1. The Bert-based models showed state-of-the-art performance in the latest disfluency detection studies, which is the most common approach [1,2]. As our work’s main contribution is to create a dataset for multi-turn discontinuity cleanup and to build the basic common models for a benchmark, we focused on checking the most widely-used models’ results instead of pursuing the highest performance.\n\n2. The GPT4 had not been released yet when we started to design and conduct our large-scale modeling and evaluation experiments. \n\nWe thank the reviewer for sharing their insights and results of giving the conversation instance to GPT4. This interesting and promising finding also potentially indicates the validity of our proposed novel dataset and task. \n\nCurrently, we believe that using Bert-based models is sufficient to serve our goal: illustrating our novel task’s solvability and difficulty compared with classical approaches. However, we are also interested in exploring more modeling innovations with LLMs on our contributed data and task in future work. We will describe the reviewer’s finding and encourage this promising direction in our revised paper version too.\n\nReferences:\n\n[1] Rocholl, Johann C., et al. \"Disfluency detection with unlabeled data and small BERT models.\" arXiv preprint arXiv:2104.10769 (2021).\n\n[2] Morteza Rohanian and Julian Hough. 2021. Best of Both Worlds: Making High Accuracy Non-incremental Transformer-based Disfluency Detection Incremental. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (ACL-IJCNLP), pages 3693–3703, Online. Association for Computational Linguistics.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1 This paper is well written and easy to follow\n2 The labelling schema is well designed and the problem of disfluency in the spoken dialogues is an interesting topic to explore.\n3 The authors build a fairly solid baseline for other works to follow-up", "reasons_to_reject": "1 Lack of analysis by using the latest LLMs\nMaybe this problem can be very-well addressed by the LLMs, the authors should post a result by using LLM. I give the conversation instance in table1 to gpt 4, and show it a few examples of discontinuity, and it did quite a good job.", "questions_to_authors": "na", "paper_topic_and_main_contributions": "This paper proposes a multi-turn cleanup dataset to remove the numerous discontinuity during the spoken dialogues. Multi-turn cleanup is sort of disfluency detection on the dialogue level instead of utterance level. The paper carefully designs a schema to collect the high-quality data. This dataset might be quite helpful to the study of disfluency detection in the dialogue domain", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "Na"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper present multi-turn spoken conversation transcript cleaning process. They label switchborad corpus with proposed annotation shceme.

Pros:
- paper presents a dataset and baseline experiments
- labelling schema is well designed

Cons:
- it is unclear if dataset is better for downstream tasks
Decision: Accept-Main