DISCO: A Large Scale Human Annotated Corpus for Disfluency Correction in Indo-European Languages
Authors: Vineet Bhat
Preethi Jyothi
Pushpak Bhattacharyya
Venue: EMNLP 2023 Findings (2023)
Keywords: Disfluency Correction
Machine Translation
Dataset
TL;DR: In this paper, we present the DISCO Dataset for Disfluency Correction in 4 select Indo-European Languages.

Abstract: Disfluency correction (DC) is the process of removing disfluent elements like fillers, repetitions and corrections from spoken utterances to create readable and interpretable text. DC is a vital post-processing step applied to Automatic Speech Recognition (ASR) outputs, before subsequent processing by downstream language understanding tasks. Existing DC research has primarily focused on English due to the unavailability of large-scale open-source datasets. Towards the goal of multilingual disfluency correction, we present a high-quality human-annotated DC corpus covering four important Indo-European languages: English, Hindi, German and French. We provide extensive analysis of results of state-of-the-art DC models across all four languages obtaining F1 scores of 97.55 (English), 94.29 (Hindi), 95.89 (German) and 92.97 (French). To demonstrate the benefits of DC on downstream tasks, we show that DC leads to 5.65 points increase in BLEU scores on average when used in conjunction with a state-of-the-art Machine Translation (MT) system. We release code to run our experiments along with our annotated dataset here.

DISCO: A Large Scale Human Annotated Corpus for Disfluency Correction in Indo-European Languages
Vineet Bhat, Preethi Jyothi, Pushpak Bhattacharyya Indian Institute of Technology Bombay, India
Abstract
making it easier for automatic MT systems to trans- late these utterances. Moreover, the removed erro- neous words are not translated which makes the output translation fluent containing all semantics from the source sentence. Table 1 illustrates ex- amples where Google MT produces disfluent and difficult-to-read English translations of disfluent sentences in 3 languages Hindi, German and French, establishing the need for DC.
Disfluency correction (DC) is the process of removing disfluent elements like fillers, repe- titions and corrections from spoken utterances to create readable and interpretable text. DC is a vital post-processing step applicd to Auto- matic Speech Recognition (ASR) outputs, be- fore subsequent processing by downstream lan- guage understanding tasks. Existing DC re- search has primarily focused on English due to the unavailability of large-scale open-source datasets. Towards the goal of multilingual dis- flueney correction, present high-quality human-annotated DC corpus covering four important Indo-Furopean languages: English, Hindi, German and French. We provide exten- sive analysis of results of state-of-the-an DC models across all four languages obtaining Fl scores of 97.55 (English). 94.29 (Hindi), 95.89 (German) and 92.97 (French). To demonstrate the benefits of DC on downstream tasks, show that DC leads to 5.65 points increase in BLEU scores on average when used in con- junction with state-of-the-art Machine Trans- lation (MT) system. We release code to run our experiments along with annotated dataset here!.
Disfluent Sentence
Google MT output
बाब में शमम अहरह रमाट बोस में रोका उभशl रলिं टाडमर रिपम करर
rumning in watch ahhh resume running
paused timer in smart watch
je veux je veux euh en- registrer une une euh vidéo instagram
I want I want uh record a uh video on insta- gram
ich brauche eine fahrt äh eine fahrt zum bahn- hof in einer stunde
need ride er ride the train station in an hour
Table 1: English translations produced by Google MT for disfluent sentences in Hindi, French and German. All disfluent words are marked in red.
1 Introduction
Humans often think and speak simultaneously in conversations, introducing erroncous words in ut- terances (Gupta et al., 2021). These words do not contribute to semantics of sentence and hence can be removed to create fluent and easy-to-interpret utterances. Disfluency Correction (DC) is defined the removal of such disfluent elements from spo- ken utterances (Shriberg, 1994).
Previous work in DC has leveraged variety of machine learning models for removing disfluent utterances from text (Ostendorf and Hahn, 2013b; Rasooli and Tetreault, 2015; Zayats et al., 2016b). However, data in DC is scarce, limiting the use of large transformer models. Switchboard (Godfrey et al., 1992), the most extensively available open- source DC corpus, contains English spoken utter- ances with only 5.9% disfluent words in the entire dataset (Charniak and Johnson, 2001). Synthetic Data Generation (SDG) has emerged a viable solution to the data scarcity problem (Passali al., 2022; Kundu et al., 2022). However, SDG can be challenging as needs expert grammatical knowl-
Motivation: Apart from making sentenees read- able and interpretable, DC also helps downstream natural language processing tasks like Machine Translation (MT) (Rao et al., 2007; Wang et al., 2010). Removing disfluencies shortens sentences,

---

Disfluency
Description
Example
Type Filler
Words like uhh, err, uhmm that are often uttered to retain turn of speak-
EN: Write a message to um Sarah.
DE: Fortsetzen ähm meines Lauftrainings.
ing. Each language has a differ-
FR: Montre euh mes applications.
ent set of filler words commonly ut- tered.
HI: मरा 3 परसरट पिटीितत में रेक करी
Repetition
Consists of words or phrases that are repeated in conversational speech
EN: Add this number to to my contacts.
DE: ein Instagram-Fott machen machen. FR: Enregistre mes 400 calories enregistre. HI: बया g4 हारसपर का हासरपटीी का पको नोट कमा सकके हो?
Correction
Disfluencies that consist words incorrectly spoken and immediately corrected with fluent phrase
EN: Get me the order my order status on the desk chair ordered from Overstock.
DE: HD Video auf aufnehmen.
FR: Reprendre l'exercice d'étirem
d' d'étirement
HI: रा रīğ रंहली का मरीग पदी
False Start
Examples where the speaker changes their chain-of-thought mid sentence to utter completely different fluent phrase
EN: In an email let's email Tom Hardy about Saturday's video shoot.
DE: Facebook uh Jahr Facebook bitte.
FR: Envoi de envoi du SMS à maman.
HI: कhटी 3म4 आागा बी काड मेशर रो बहाअा
Fluent
Examples which do not contain any disfluent words or phrases
EN: Can you make a note for Johnny that says
dinner at eight my laptop?
DE: Nummer zu Kontakten hinzufügeen..
FR: Je veux j j'aimerais ouvrir TikTok.
HI: बया आाप पोसीमडटट नियन पाक को डिन कर सकके ?
Table 2: Types sentences observed in the DISCO corpus. All disfluencies are marked in red; EN-English, DE- German, FR-French, Hl-Hindi.. Examples in languages other than English, with their corresponding gloss and transliteration can be found in Appendix
edge and the data created can often fail to mimic complex disfluencies encountered in real-life dia- logues (Gupta et al., 2021).
fluent text utterance pairs in 4 languages: En- glish, Hindi, German and French with exten- sive data analysis (Section 3.4).
Hence there a dire need to develop DC datasets with utterances from real-life conversa- tional situations. Existing datasets have focused on increasing the available data in English. This paper presents high-quality DC corpus in English and widely spoken languages like Hindi, German and French. Our dataset significantly expands the available data in English and Hindi. To the best of our knowledge, we are the first create an open- source DC corpus for German and French?. Our contributions are:
Experimenting with various statc-of-the-art techniques for DC ranging from traditional ML models to large transformers (Table 5). Our best models (fine-tuned multilingual transformers) achieve an Fl score of 97.55 (English), 94.29 (Hindi), 95.89 (German) and 92.97 (French). Our results in English and Hindi competitive with other approaches, but do report direct improvement due the different testing datasets used.
Improving BLEU score of state-of-the-art MT system by 5.65 points in Hindi-English and German-Lnglish language pairs after au- tomatic disfluency removal from source sen-
1. A laben-labeled dataset of 1 disfluent-
Althouggh Cho et al. (2014) annotated the KIT lecture cor- pus (Stüker et al., 2012) for disfluencies in German, their data is not shared publically.

---

tences (Table 10). Similar analyses for other language pairs are part of our future work.
lel disfluent-fleet sentence pairs in the above four languages and English translations of fluent sen- tences in Hindi and German along with disfluency and domain labels.
2
Related Work
The study of disfluencies as spoken language phe- nomenon was first proposed in Shriberg (1994). DC has been established as a vital post-processing task for ASR transcripts (Rao et al., 2007; Wang et al., 2010). Although earlier DC systems were based on translation methods (Honal and Schultz, 2003), current research covers two additional methods: parsing-based and sequence tagging- based techniques. Translation-based methods use a noisy channel approach towards DC hypothesiz- ing that disfluent sentences are fluent sentences with noisy elements (Jamshid Lou and Johnson, 2017; Johnson and Chamnik, 2004; Zwarts and Johnson, 2011). Parsing-based methods use tech- niques such as dependency parsing to predict syn- tactic structure of utterance along with disflu- ent elements (Rasooli and Tetreault, 2015; Hon- nibal and Johnson, 2014; Wu al., 2015). Se- quence tagging methods work well for disfluency removal from real-life spoken utterances, assign- ing disfluent/fluent label to every word in the sen- tence (Hough and Schlangen, 2015a; Ostendorf and Hahn, 2013a; Zayats et al., 2016a; Chen et al., 2022). Language clues and part-of-speech tags based systems have also been explored for DC (Bove, 2008; Christodoulides al., 2014). There is a notable gap literature regarding real data an- notation in DC, with Switchboard (Godfrey et al., 1992) and Salesky et al. (2018) being the most ex- tensive open-source labeled datasets for English DC. Although Gupta et al. (2021) introduced dataset for disfluencies in English question answer- ing, they have not been annotated for disfluent words. Without labeled data, various zero-shot, few-shot, and multi-task learning techniques have been proposed, which train on multilingual data, creating and utilizing synthetically generated dis- fluent sentences (Wang et al., 2018; Passali et al., 2022; Kundu et al., 2022; Bhat et al., 2023). In this work, we experiment with sequence tagging meth- ods for DC.
3.1 Terminology
Shriberg (1994) defines disfluencies as a composi- tion of Reparandum, Interregnum and Repair (Fig- ure 1). Reparandum refers to words erroneously uttered by the speaker. The speaker acknowledges that previous utterance might be incorrect ing interregnum, whereas repair contains words that correct mis-spoken words. Disfluent utter- ances might consist of an interruption point- spo- ken phenomena like speech pauses. DC removes reparandum and interregnum while retaining re- pair to make the output sentence more fluent.
Internpttion Paint
Repair
Reschedule, sorry, cancel my cab ride
Reparandum
Interregnum
Figure 1: disfluent utterance in English, marked with various components of disfluencies.
We study four types of disfluencies observed in dataset: Filler, Repetition, Correction and False Start. Additionally, there some fluent sentences present in our corpus. Table 2 describes each type of sentence with some real examples from the DISCO datasct.
3.2 Data Collection Method
Goel et al. (2023) released an open-source dataset containing real-life utterances of humans with Al agents for task-oriented dialogue parsing. We ex- tract disfluent sentences and domain labels in En- glish, Hindi, German and French from this corpus. These utterances consist of human dialogues like making notes, monitoring fitness, adding new con- tacts, opening apps, etc. All sentences are shared with respective language experts for fluent sen- tence creation and disfluency-type annotation.
3 DISCO: Dataset for Disfluency Correction
3.3 Annotation Protocol and Challenges
For each language, we hired external annotators from reputed translation agencies with experience in data annotation. They were asked to create flu- ent sentences corresponding to disfluent ufterances
This section analyzes the DISCO corpus, created with the help of English, Hindi, German and French language experts. DISCO contains paral-

---

we follow a sequence tagging approach towards DC, the annotators were asked to only remove disfluent words from utterances without changing word order or correcting original words/phrases. along with disfluency type labels. Lach annotator was paid competitively based on current market standards (approximately $ 0.018 per word). Since
Lang
Mean of sentences
length disfluent
Mean of fluent sen- tences
length
En
9.19 2.85
7,45 2.59
Hi
10.18 + 3.60
8.24 3.12
De
7.25 3.12
5.71 2.84
Fr
7.42 + 3.05
6.08 + 2.87
Due to budget constraints, we could not utilize the entire dataset in German and French from Goel al. (2023). However, we carefully select sen- tences these languages sufficiently cover all disfluency types with varied length and complex- ity utterances. Table summarizes the total amount of data created and the amount of disflu- ency present in the corpus.
Table 4: Average length of disfluent and fluent utter- ances in the DISCO corpus for each language: En- English, Hi-HHindi, De-German, Fr-French.
Our corpus also contains a good distribution of sentences across various task domains. Readers are urged refer Appendix B for the domain- level distribution and other important plots pertain- ing to the corpus.
Lang No. of No. % dis-
sentence of fluent pairs
words
words
3.5 Helper Datasets
En
3479
31994
18.99
We also use some helper datasets, extracting unla- beled sentences enable few shot learning-based experiments on DISCO.
Hi
3180
32435
18.99
De 3096
22451
20.93
Fr
3005
22489
17.72
LARD: Contains synthetically generated English disfluent sentences using rule-based disfluency in- jection in fluent sentences (Passali et al., 2022).
Table 3: Total count disfluent-- fluent pairs DISCO and percentage of disfluency present; En-Ennglish, Hi- Hindi, De-German, Fr-French.
Samanantar: Consists of 49.7 million parallel sentences between English and 11 Indic languages (Ramesh et al., 2021). Source sentences were col- lected across many domains such as newspapers, government public archives, Wikipedia, etc. The corpus consists of fluent sentences, and we only use Hindi sentences for our experiments.
Since for every language, only one annotator created fluent sentences and disfluency type labels, ensuring high quality data was very important. We strongly encouraged the annotators to flag all dubi- ous instances, after which the authors take major- ity vote of retaining/removing doubtful disfluent words using high quality translation tools and sub- ject knowledge wherever necessary. Flagged amples and our reasoning for specific annotations have been discussed in Appendix A.
GermEval 2014: Consists of 31K German flu- sentences collected from Wikipedia and vari- ous news corpora (Benikova al., 2014). Origi- nally used for Named Entity Recognition, we uti- lize unlabeled sentences from the train split.
3.4
Key Statistics
DiaBLa: Released by Bawden et al. (2021), this corpus consists of 5700+ sentence pairs for Fnglish-Freech MT. The dataset is curated from written and informal interactions between native speakers in both languages.
The DISCO corpus is carefully created ensure healthy representation of various disfluency types and complexity of sentences. Table 4 describes av- erage length of disfluent and fluent sentences for each language. Our analysis shows that sim- ilar context, commands to AI agents are shorter in German and French than in English and Hindi. The standard deviation of the disfluent sentences demonstrates that the dataset also contains longer utterances, more than words long, in each lan- guage that are relatively difficult correct. We showease the distribution of disfluent sentences across disfluency types in figure 2.
Dataset Evaluation
This section describes the experiments we perform to evaluate the DISCO corpus. Our evaluation strategy measures the efficacy of the corpus for robust disfluency correction in wide variety of cases. Moreover, we also test the ability of our trained models to correct disfluencies for improv- ing downstream machine translation.

---

Distribution of disfluent sentences English
Distribution of disfluent sentences in Hindi
1750
1600
1500
1400
1200
Frequency
1000
Frequency
1000
800
750
600
400
250
200
Repetition Correction Start
Repetition
Start
Disfiuency Types
Disfiuency Types
Distribution of disfluent sentences in German
Distribution of disfluent sentences in French
1400
1400
1200
1200
Frequency
1000
Frequency
1000
800
600
400
400
Repetition
Start
Repetition
Start
Disfluency Types
Disfiuency Types
Figure 2: Distribution of sentences across disfluency types for all four languages in DISCO.
4.1
Data Processing
models.
All parallel sentence pairs are passed through punctuation removal module to reduce the number of tokens for classification. As per the structure of disfluencies described in section 3.1, we consider fluent terms to always follow disfluent terms in an utterance. Disfluent ufterances are marked with the positive label (1) and fluent utterances with the neutral label (0) (Kundu et al., 2022).
4.2.2 Transformer Baselines
Transformers (Vaswani et al., 2017) are large and powerful neural networks capable of learning com- plex text representations for many downstream tasks. We experiment with three multilingual transformers: mBERT (Devlin et al., 2019), XLM- R (Conneau et al., 2020) and MuRIL (Khanuja et al., 2021). Finctuning for sequence tagging is performed by adding classification head (on top of these transformers) that performs sub-word level binary prediction. Prediction of word to be disfluent/fluent is the prediction of the first sub- word to be disfluent/fluent.
4.2 Baseline Models
We use combination of smaller ML models, larger transformer models and transformers with adversarial training. All models are trained on an 80: 10:10 train:valid:test split for each language.
4.2.3
Transformer with Adversarial Training (Seq-GAN-BERT)
4.2.1 ML Baselines
Previous work has shown the efficacy using Conditional Random Fields (CRFs) and Recur- rent Neural Network (RNN) based techniques for token classification in DC (Ostendorf and Hahn, 2013b; Hough and Schlangen, 2015b). These mod- els require fewer labeled data and are ideal for low-resource domain-specific training (Simpson al., 2020). Token-level features from a powerful multilingual transformer, XLM-R (Conneau et al., 2020), were used for finetuning the CRF and RNN
In low-resource settings, adversarial training helps transformers improve the representations it learns for downstream tasks. We use the Seq-GAN- BERT model (Bhat et al., 2023), which supports adversarial training for transformers utilizing la- beled and unlabeled data for token classification- based DC. Unlabeled data is used from helper datasets specified section 3.5. We obtain the best results using MuRIL transformer as the base model in Seq-GAN-BERT.

---

4.3 Experimental Setup
Model
En
Hi
De
Fr
CRF
59.15
35.70
53.01
43.60
CRF and RNN models are trained using the Flair- NLP framework (Akbik et al., 2019) till the valida- tion cross-entropy loss saturates. We start with learning rate 0.1 and reduce it by half each time the model does not improve for three consecutive epochs. Transformer models are trained using the popular transformers package (Wolf et al., 2020). We use a learning rate of 2e-5 and a weight decay 0.01. All transformer models are trained for 40 epochs using the Adam optimizer (Kingma and Ba, 2014).
RNN
83.28
69.96
81.11
82.50
mBERT
96.94
88.08
93.70
92.97
XLMR
95.95
91.31
95.89
92.35
MuRIL
96.65
94.29
92.00
92.48
Seq-GAN- BERT
97.55
93.71
88.95
86.23
Table 5: Results in DC for each language. For Seq-GAN-BERT, we report best results with helper datasets (section 3.5): English (En): LARD, Hindi (Hi); Samanantar, German (De): GermEval 2014, French (Fr): DiaBLa. Since we are the first to create DC cor- pus in German and French and with existing English and Hindi datasets being vastly different its proper- ties and sources, we do not provide zero-shot metries of our best models on other datasets.
4.3.1 Hardware support
All ML baselines were trained with A 100 GPUs provided by Google Colab. Transformers were trained with one NVIDIA GeForce RTX P8-11GB GPU per experiment.
5 Results and Analysis
strongly urged to refer to Appendix for domain- level analysis of DC results. Although existing DC datasets are of diverse domains, our experiments show that models trained on DISCO outperform test sets from other DC datasets (Appendix D).
We thoroughly analyse all experiments performed in DC. This section also discusses some case stud- ies highlighting strengths and weaknesses of our best models. Our experiments in analyzing the im- pact of DC on MT provides interesting linguistic insights into the phenomenon of disfluencies.
Type Filler
En
Hi
De
Fr
5.1 Disfluency Correction
99.78
100.00
99.37
98.72
All results are reported using the F1 score met- ric (Jamshid Lou and Johnson, 2017; Passali et al., 2022). Combined results across all four languages are described in table 5. As the complexity of mod- els increases, the overall accuracy also increases. Transformer architectures perform better than CRF and RNN-based models consistently. In each lan- guage. the best models produce 90+ F1 scores on blind test sets, indicating that our corpus suc- cessfully solves the data scarcity problem. As ex- pected, Fl scores of multilingual transformers close due to similiar range of parameters that are fine-tuned for token classification based DC.
Repetition
98.48
92.81
97.13
98.58
Correction
91.72
91.54
98.48
93.94
False Start
97.19
85.04
90.91
98,00
Fluent* 66.67
91.03
96.30
57.14
Fl scores for every disfluency type in each lan- guage using our best DC model. *We report F1 score of fluent class here because for dislluent class, true pos- itives is equal to zero.
Table 6:
Table 7 discusses some important case stud- ies containing inferences produced by our model on unseen test sentences. Our models accurately correct complex cases such as multiple disfluen- cies and change in thought/speech plan. How- ever, it also over-corrects clarifications and treats it as correction to remove significant meaning. We observe that multi-words, critical linguistic phenomenon in Indian languages, often over- corrected to simplify the text. More case studies appear in Appendix E along with gloss and English transliteration for non-roman text.
Performance across disfluency types de- scribed in table 6. We observe that the model per- forms poorly for fluent sentences in English and French due to fewer samples in the test set. In Hindi and German, false starts the most diffi- cult disfluencies correct. Further examination reveals that our model often under-corrects longer false starts, especially in the presence of other dis- fluencies like fillers. Our model performs robustly across all domain types of utterances. Readers are

---

Lang Type Disfluent Sentence
Prediction
Comments
En C
is etrading mean ameri- trade the top trading app currently
is ameritrade the top trading app currently
Correctly identifies change in main content
R
add 20minute 2mile walk walk to myfitnesspal
add 2mile walk to myfitnesspal
Correctly removes repeated word walk but mistakes 2mile as correction to 20minute
De
R,C
trage
die
übung
die trage die laufübung ein
Model correctly detects cor- rection of übbng (exercise) to laufübung (rumning exercise)
laufübung ein
FS zeig mir wie sag mir wie es den aktienpreisen bei der nasdaq gestern ging
zeig mir wie sag mir wie es den aktien- preisen bei der nasdaq gestern ging envoyez mon message audio
Model fails remove false started phrase zeig mir wie it has an independent meaning
Fr
F,R envoyez mon message euuh mon message audio
Model correctly identifies that the user repeats the phrase to de- note an audio message. Model incorrectly thinks hd is disfluent term and not the abbre- viation of High Definition Chain in thought process de- tected & corrected
F
enregistrer une vidéo hd avec hmm instagram
enregistrer une vidéo
avec instagram
Hi
C
कि gah नही कीससर कार दी कार राम की शिररी बे रिंद अोता पर
दि करिल बरर दी कार शाम की पिररीी के शिरर अोला पर
R ममोन पर आाय यटट आाय पंटा पहरट रमिा रा करर टी अपी के अrीe
पाोन पर आाеा पाटा पारе रनिग qरe कार दी Rn
Model removes repetition but also over corrects the multi-word phrase: अप4ी के अपभी
Table 7: Inference examples from DC models; En-English, De-German, Fr-French, Hi-Hindi; F-Filler, R- Repetition, C-Correction and FS-False Start.
Lang
Setup
F
R
C
FS
els important downstream NLP task.
Pair
The Ground Truth (GT) translations for Hindi- English and German-English were created by re- spective language experts. We use the sacrebleu package (Post, 2018) calculate BLBU scores be- tween: Tl (Translations without DC) and GT; T2 (Translations with Automatic DC) and GT; and T3 (Translations with Human DC) and GT. Table 10 summarises our results in both language pairs. DC improves downstream MT for Hindi-English by 6.44 points and for German German-English by 4.85 points in BLEU score. We also observe that human DC outperforms automatic DC, highlighting scope of improvement of DC models.
Hi-En
No DC
31.79
41.45
29.37
38.67
ADC
42.34
47.92
35.94
45.54
HDC
42.96
47.69
38.71
45.23
De-
No DC
37.95
39.36
38.95
57.31
En
ADC
51.20
48.50
50.76
67.35
HDC
51.20
48.63
51.32
68.40
Table 8: Effect of each disfluency type and its re- moval on downstream MT for Hindi-English (Hi-Fn) and German-English (De-En) language pairs. F-Filler, R-Repetition, C-Correction and FS-False Start.
Table 8 shows that translation BLEU score im- proves for every disfluency type after DC. More- over, in repetition and false starts, the automatic removal of DC slightly outperforms Human DC. The most significant improvement in BLEUU score is observed fillers, with the lowest improvement in corrections. Our models also improve the scores
5.2 Impaet of Disfluency Correction on Downstream Machine Translation
We strong baseline NLLB MT system (Costa- jusså et al., 2022) compare English translations produced with and without disfluency removal (Appendix F) to understand the impact of DC mod-

---

Lang Disfluent Pair
Predicted
Translations
Observations
Sentence
Fluent Sentence
(ADC)
De-
30 nein
50 50 minuten
Tl: 30 no 50 minutes start jog- ging
Correction from 30 to minutes is identified by DC which leads to fluent translations T2 and T3 ADC mistakenly re- moves both utterances of weitwinkel (wide angle) leading downstream translation error TI difficult to inter- pret due to the presence of translated false start phrase.
En
joggen joggen
minuten
starten
T2: Start running for 50 minutes
starten
T3: Start jogging for 50 minutes.
mache eine mache weitwinkel video also eine weitwinkel
ein
T1: So make wide angle video
T2: Make video
T3: Make wide angle video
video
Hi-
बाब में शमिा साटे बोब अहटट साद
T1: Running in the Watch Ahh Smart Watch Restart the Running Time that has been blocked in the Watch
En
रोका T शनिग टाडमR रलपम करी
बां में रोका
द शमि
टाडमर रिपम
T2: Resume the running timer that is blocked in the smartwatch
करी
T3: Resume stopped running
timer in smart watch.
मील काो टोडना yक अ
5 मोश का
T1: Running 5 miles a a a a just
Due to incomplete disflu- ency correction, dubious translation (bus or stop) of the following word causes an issue: कस
टोडना पक① बस
stop now T2: 5 miles running a bus now
अ अ बस अन
अह कद करर दो
पदद कार टी
stop
T3: Stop running 5 miles now
Table 9: Examining some examples where disfluencies impact machine translation output for German-English (De- En) and Hindi-English (Hi-En) language pairs
Setup
Hi-En
De-En
Model (Dataset)
BLEU Score
MT without DC
36.19
37.08
MuRIL (DISCO)
42.63
MT with ADC
42.63
41.93
MuRIL (Kundu et al., 2022)
38.97
MT with HDC
43.52
42.10
Table 11:
Comparing the performance of DC systems trained on different datasets on the Hindi English
Table 10: Effect of DC on downstream MT for Hindi- English (Hi-En) and German-English (De-En) lan- guage pairs. ADC: Automatic Disfluency Correction, HDC: Human Disfluency Correction
DISCO MT improvement task when used with state- of-the-art MT system (NLLB)
5.2.1 Case Study: translations with and without disfluency correction
Examining a few
across all domains, as described in Appendix F.
Table discusses some interesting cases where dis- fluencies impact the English translation of Hindi and German sentences. Although removing disflu- encies in most cases helps MT, there are few exam- ples where DC leads to worse output.
We also compared the downstream MT im- provement caused by large transformer (MuRIL) trained separately
both DISCO and (Kundu al., 2022) for Hindi DC followed by downstream Hindi English translation using NLLB, Table 11 highlights that MuRII. trained on DISCO leads to 3.66 BLEU score improvement relative to the base- line.
Conclusion and Future Work
This paper introduces the DISCO datasct for disflu- ency correction in four widely spoken languages:

---

English, Hindi, German and French. Our work highlights the importance of large-scal projects in NLP that scale the amount of labeled data avail- able. Spoken interactions between humans and AT agents are riddled with disfluencies. Flliminating disfluencies not only improves readability of utter- ances but also leads to better downstream trans- lations. Our dataset, which consists of roughly 3000 parallel disfluent-fluemt sentences in each lan- guage, significantly reduces the data scarcity prob- lem in DC. This allows training of large trans- former models to correct spoken disfluencies from written transcripts with high accuracy. Lack of conversational translation datasets has led to most MT systems trained on fluent text. Our experi- ments show that such models if used in conversa- tional settings do not perform well. By adding DC model in the pipeline, which is often a smaller model with an incremental increase in latency, one can improve the downstream translations out- putted by an MT system that does not adjust to con- versational phenomena. Moreover, our dataset in German English and Hindi English can also be used to finetune conversational MT models.
Limitations
Our work consists two limitations. Firstly, since our annotation process consisted of one annota- tor for cach language, we could not report met- rics such as inter-annotator agreement or Cohen's kappa to prove the validity of our dataset. How- ever, since DC is relatively more straightforward task and consists of only removing disfluent words from spoken utterances, the authors were able to verify many samples as a part of their analysis. Moreover, the structure of disflueccies helps us recognize disfluency types easily. We have also provided few flagged cases where annotators dis- cussed their queries with us and how we resolved them.
Secondly, we do not compare trained models on DISCO with other datasets due to varied do- main of existing datasets. We found that existing datasets like Switchboard (Godfrey et al., 1992), LARD (Passali et al., 2022) and Kundu et al. (2022) all consisted of utterances from very di- verse data sources. However we include experi- ments in Appendix D that highlight the robustness of models trained DISCO.
Future work lies in experimenting with better ML models for sequence tagging-based DC sup- porting multilingual training. These should also incorporate linguistic features like reparandum, in- terregnum and repair. Multimodal DC presents a promising direction as it has the capability of us- ing both speech and text features for correction tasks (Zhang et al., 2022). Additionally, trained DC models must be evaluated using diverse sam- ples from various domains and dialects. Special efforts must be made to collect disfluent speech transcripts to be annotated and trained for DC in other low-resource languages.
Ethics Statement
This work publishes a large scale human anno- tated dataset for disfluency correction in 4 Indo- European languages English, Hindi, German and French. We have taken all steps to ensure that the data is collected and annotated using all eth- ical means. The source sentences of our dataset are extracted from Goel ct al. (2023) which release the data using the CC by 4.0 license, allowing us to remix, transform, and build upon the material for any purpose. We also follow a stringent data annotation protocol with consent from the annota- tors and ensuring they aware of the risks sociated with data creation. We also mention the compensation paid to them for their contribution in section 3.3. Since this project is not sponsored by federal body, we do not use the IRB approval for our work. However, attention is paid to the quality of our dataset with flagged cases discussed extensively with annotators to ensure appropriate resolution (Appendix A). A thorough and exten- sive analysis of our corpus performed, details of which are provided in section 3. All datasets used in conjunction with our corpus are open-source and cited appropriately in section 3.5. We understand that the dataset might have some mistakes, and we
7
Acknowledgements
We would like to thank the anonymous reviewers and area chairs for their suggestions to strengthen the paper. This work was done as part of the Bahubhashak Pilot Project on Speech to Speech Machine Translation under the umbrella of Na- tional Language Technology Mission of Ministry of Electronics and IT, Govt. of India. We would also like thank the project managers, internal and external language translators the Computa- tion for Indian Language Technology (CFILT) IIT Bombay.

---

will continuously work on monitoring and resolv- ing such issues once the corpus is published for open-source research. Our robust results across do- mains and types of sentences ensure that changes to the dataset do not pose any technical issues or risks to both developers and model users.
Resources and Evaluation (LREC'14), pages 1554- 1559, Reykjavik, Iceland. European Language Re- sources Association (ELRA).
George Christodoulides, Mathieu Avanzi, and Jean- Philippe Goldman, 2014. DisMo: morphosyn- tactic, disfluency and multi-word unit annotator. an evaluation on corpus of French spontancous and read speech. In Proceedings of the Ninth Interna- fional Conference on Language Resources and Eval- uation (LREC'14), Reykjavik, Iceland. European Language Resources Association (ELRA).
References
Alan Akbik, Tanja Bergmann, Duncan Blythe, Kashif Rasul, Stefan Schweter, and Roland Vollgraf. 2019. FLAIR: An easy-to-use framework for state-of-the- art NLP. In NAACL 2019. 2019 Annual Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations), pages 54 59.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zeutl- moyer, and Veselin Stoyanov. 2020. Unsupervised cross-linguss representation learning at scale. In Pro- ceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 8440- 8451, Online. Association for Computational Lin- guístics.
Rachel Bawden, Eric Bilinski, Thomas Lavergne, and Sophie Rosset. 2021. Diabla: A corpus of bilingual spontaneous written dialogues for machine transla- tion. Language Resources and Evaluation, 55:635 660.
Marta Ruiz Costa-jussă, James Cross, Onur eCelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffer- nan, Flahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wen- zek, Alison Youngblood, Bapi Akula, Loic Bar- rault, Gabriel Mcjia Gonzalez, Prangthip Hansanti, John Hoffman, Semarley Jarrett, Kaushik Ram Sadagopan, Dirk Rowe, Shannon I.. Spruit, C. Tran, Pierre Yves Andrews, Necip Fazil Ayan, Shruti Bhosale, Sergey Edunov, Angela Fan, Cynthia Gao, Vedanuj Goswami, Francisco Guzm' 'nn Philipp Koehn, Alexandre Mourachko, Christophe Rop- ers, Safiyyah Saleem, Holger Schwenk, and Jeff Wang. 2022. No language left behind: Scal- ing human--cmtern machine translation. ArXiv, abs/2207.04677 04677.....
Darina Benikova, Chris Biemann, and Mare Reznicek. 2014. NoSta-D named entity annotation for German: Guidelines and dataset. In Proceedings of the Ninth International Conference
Language Resources and Evaluation 14), pages 2524-2531, Reyk- javik, Iceland. European Language Resources Asso- ciation (ELRA).
Vineet Bhat, Preethi Jyothi, and Pushpak Bhat- tacharyya. 2023, Adversarial training for low- resouree disfluency correction. In Findings tfe As- sociation for Computational Linguistics: ACL 2023. pages 8112-88111 8122, Toronto, Canada. Association for Computational Linguistics.
Rémi Bove. 2008. A tagged corpus-based study for re- peats and self-repairs detection in french transcribed speech. In Text, Speech and Dialogue, pages 269- 276, Berlin, Heidelberg. Springer Berlin Heidelberg.
Jacob Devlin, Ming-Was Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minncapolis, Minnesota. Associ- ation for Computational Linguistics.
Eugene Charniak and Mark Johnson. 2001. Edit detec- tion and parsing for transcribed speech. Second Meeting of the North American Chapter of the Asso- ciation for Computational Linguistics.
Angelica Chen, Vicky Zayats, Daniel Walker, and Dirk Padfield. 2022. Teaching BERT to wait: Balancing accuracy and latency for streaming disfluency detec- tion. In Proceedings of the 2022 Conference of the North American Chupter of the Association for Com- putational Linguisties: Human Language Technolo- gies, pages 827 838, Seattle, United States. Associ- ation for Computational Linguistics.
John J. Godfrey, Edward Holliman, and McDaniel. 1992. Switchboard: telephone speech corpus for re- search and development. (Proceedings)) ICASSP-92: 1992 IEEE International Conference on Acousties, Speech, and Signal Processing, 1:517-520 vol.1.
Rahul Goel, Waleed Ammar, Aditya Gupta, Siddharth Vashishtha, Motoki Sano, Faiz Surani, Max Chang, HyunJeong Choe, David Greene, Kyle He, Rattima Nitisaroj. A. A. Trukhina, Shachi Paul, Pararth Shah, Rushin Shah, and Zhou Yu. 2023. Presto: A multi- lingual dataset for parsing realistic trientd di- alogs. ArXiv, abs/2303.08954,
Eunah Cho. Sarah Fünfer, Sebastian Stüker, and Alex Waibel, 2014. A corpus of spontaneous speech in lectures: The KIT lecture corpus for spoken lan- guage processing and translation. In Proceedings of the Ninth International Conference on Language

---

Aditya Gupta, Jiacheng Xu, Shyam Upadhyay, Diyi Yang, and Manaal Faruqui. 2021. Disfl-QA: A benchmark dataset for understanding disfluencies in question answering. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 3309-3319.
Mari Ostendorf and Sangyun Hahn. 2013b. A sequen- tial repetition model for improved disfluency detec- tion. In Interspeeck 2013, pages 2224.6628. ISCA.
Tatiana Passali, Thanassis Mavropoulos, Grigorios Tsoumakas, Georgios Mcditskos, and Stefanos Vrochidis. 2022. LARD: Large-scale artificial dis- fluency generation. In Proceedings of the Thir- teenth Language Resources and Evaluation Confer- ence, pages 2327-2336, Marseille, France. Euro- pean Language Resources Association.
Matthias Honal and Tanja Schultz. 2003. Correc- tion of disfluencies in spontaneous speech using a noisy-channel approach. 8th European Confer- ence on Speech Communication and Technology (Eu- rospeech 2003).
Matt Post. 2018. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186- 191, Belgium, Brussels. Association for Computa- tional Linguistics.
Matthew Honnibal and Mark Johnson. 2014. Joint Incremental Disfluency Detection and Dependency Parsing, Transactions of the Association for Compu- tational Linguistics, 2:13
Gowtham Ramesh, Sumanth Doddapameni, Aravinth Bheemaraj, Mayank Jobanputra, Raghavan AK, Ajitesh Sharma, Sujit Sahoo, Harshita Diddee, Ma- halakshmi J, Divyanshu Kakwani, Navneet Kumar, Aswin Pradeep, Srihari Nagaraj, Kumar Deepak, Vivek Raghavan, Anoop Kunchukuttan, Pratyush Kumar, and Mitesh Shantadevi Khapra. 2021. Samanantar: The largest publicly available parallel corpora collection for 11 indic languages.
Julian Hough and David Schlangen. 2015a. Recur- rent neural networks for incremental disfluency de- tection.
Julian Hough and David Schlangen. 2015b. Recurrent neural networks for incremental disfluency detection. In Proc. Interspeech 2015, pages 849-853.
Paria Jamshid Lou and Mark Johnson. 2017. Disflu- ency detection using a noisy channel model and a deep neural language model. In Proceedings of the 55th Annual Meeting of the Association for Compu- tational Linguistics (Volume 2: Short Papers), pages 547-553, Vancouver, Canada. Association for Com- putational Linguistics.
Sharath Rao, Ian Lane, and Tanja Schultz. 2007. Im- proving spoken language translation by automatic disfluency removal: evidence from conversational speech transcripts. In Proceedings of Machine Translation Summit XI: Papers, Copenhagen, Den- mark.
Mark Johnson and Eugene Charniak. 2004. A tag- based noisy channel model of speech repairs. In Pro- ceedings of the 42nd Annual Meeting on Association for Computational Linguistics, ACL '04, page 33 es. USA. Association for Computational Linguistics.
Mohammad Sadegh Rasooli and Joel R. Tetreault. 2015. Yara parser: A fast and accurate dependency parser. Computing Research Repository, arXiv: 150.00633. Version 2.
Elizabeth Salesky, Susanne Burger, Jan Nichues, and Alexander H. Waibel. 2018. Towards fluent transla- tions from disfluent speech. 2018 8 IEFF Spoken Lan- guage Technology Workshop (SI.T), pages 921-926.
Simran Khanuja, Diksha Bansal, Sarvesh Mehtani, Savya Khosla, Atreyee Dey, Balaji Gopalan, Dilip Kumar Margam, Pooja Aggarwal, Rajiv Teja Nagipogu, Shachi Dave, Shruti Gupta, Subhash Chandra Bose Gali, Vish Subramanian, and Partha Talukdar. 2021. Muril: Multilingual representations for indian languages.
Elizabeth Shriberg. 1994. Preliminaries theory of speech disfluencies.
Edwin Simpson, Jonas Pfeiffer, and Iryna Gurevych. 2020. Low resouree sequence tagging with weak la- bels. Proceedings of the AAAI Conference on Artifi- cial Intelligence, 34(05))8862-8869.
Diederik P. Kingma and Jimmy Ba. 2014. Adam: method for stochastic optimization. CoRR, abs/1412.
Rohit Kundu, Preethi Jyothi, and Pushpak Bhat- tacharyya. 2022. Zero-shot Disfluency Detection for Indian Languages. In Proceedings of the 29th In- lernational Conference on Compulational Linguis- ties, pages 4442 4454, Gyeongju, Republic of Ko- rea. International Committee on Computational Lin- guistics.
Sebastian Stüker, Florian Kraft, Christian Mohr, Teresa Herımann, Eunah Cho, and Alex Waibel. 2012. The KIT lecture corpus for speech translation. In Pro- ceedings of the Fighth International Conference on Language Resources and Evaluation (LREC' 12), pages 3409 3414, Istanbul, Turkey. European Lan- guage Resources Association (ELRA).
M. Ostendorf and Hahn. 2013a. A sequential repeti- tion model for improved disfluency detection, Pro- ceedings of the Annual Conference of the Interna- tional Speech Communication Association, INTER- SPEECH, pages 2628..
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention Is All You Need. arXiv:1706.0376 [es]. ArXiv: 1706.03762.

---

Feng Wang, Wei Chen, Zhen Yang, Qianqian Dong. Shuang Xu, and Bo Xu. 2018. semi-supervised dis- fluency detection. In Proceedings of the 27th Inter- national Conference on Computational Linguistics. pages 1129--533,, Santa Fe, New Mexico, USA. As- sociation for Computational Linguistics.
disfluency structure and rules for token classifica- tion. In our work, we could only assign one annota- tor per language. To ensure that the data created is high quality, we had regular discussions with our annotators to resolve flagged examples. Table 12 showcases such examples with the reasoning be- hind our data annotation decisions. After verifying the quality of data created, we proceed with data analysis described in section 3.4.
Wen Wang, Gokhan Tur, Jing Zheng, and Neeip Fazil Ayan. 2010. Automatic disfluency removal for im- proving spoken language translation. In 2010 IEEE International Conference on Acoustics, Speech and Signal Processing, pages 5214-5217.
B Additional Dataset Analysis
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Rémi Louf, Morgan Fun- towicz, Joe Davison, Sam Shleifer. Patrick von Platen, Clara Ma, Yacine Jemite, Julien Plu, Can- wen Xu, Teven Le Scao, Sylvain Gugger. Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020. Transformers: State-of-the-art natural lan- guage processing. In Proceedings of the 2020 Con- ference on Empirical Methods in Natural Language System Demonstrations, pages 38 45, Online. Association for Computational Linguisties.
In this section, we provide information about the origin of the utterances that are part of the DISCO dataset. We use domain type labels from Goel el. (2023). To better understand our data, we describe each domain type under broader categorization. In each example, disfluent utterances are marked in red. The number of sentences in each domain type for each language is specified in table 13.
Processing:
Health Fitness: Sentence pairs belonging to this domain consist of utterances where the user wants to perform fitness health checkup task like recording his/her exercises, nutrition or blood sugar. Any interaction where the user discusses any fitness query can be tagged in this category. Domains such as Get health stats, Log exercise, Log nutrition, Start exercise, Stop exercise, Pause exercise and Resume exercise fall under this type.
Shuangzhi Wu, Dongdong Zhang, Ming Zhou, and Tiejun Zhao. 2015. Efficient disfluency detection with transition- based parsing. In Proceedings of the 53rd Annual Meeting of the Association for Compu- tational Linguistics and the 7th International Joint Conference on Natural Language Processing (Vol- ume 1: Long Papers), pages 495 503, Beijing, China. Association for Computational Linguistics.
Vicky Zayats, Mari Ostendorf, and Hannaneh Ha- jishirzi. 2016a. Disfluency detection using bidirec- tional Istm. pages 2523 2527.
Example Go to Fitbit and show me my um my blood sugar reading
Victoria Zayats, Mari Ostendorf, and Hannaneh Ha- jishirzi. 2016b. Disfluency detection using bidi- rectional Istm. ArXiv, mass/600.03209,
Order Status:
Sentence pairs belonging
Fan Zhang. Mei Tu, Song Liu, and Jinyao Yan. 2022. Ast error correction with dual-channel self- supervised learning. In ICASSP 2022 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7282-7286.
this domain consist of utterances where the user wants to check the status of the already placed order. The Check order status domain falls under this type.
Example - Check the status of um my Posh- mark order with FedEx
Simon Zwarts and Mark Johnson. 2011. The impact of language models and loss functions repair dis- fluency detection. In Proceedings of the 49th An- nual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 703-711, Portland, Oregon, USA. Association for Computational Linguistics.
Sentence pairs belonging to this domain consist of utterances where the user wants to perform finance task like check- ing stock market prices or getting information from a finance app. The domain Get security price falls under this type.
Finance: :
Flagged Cases in Data Creation
Data in DC is expensive and challenging to anno- tate. Language experts must not only have com-
Example want to check stock prices.
plete knowledge of the grammar and semantics of a language, but they should also be mindful of the
Bill Payment or Purchase: Sentence pairs
belonging to this domain consist of utterances

---

where the user wants complete bill pay- ment instructing the Al agent to purchase something for him/her. Domains such as Get bill, Pay bill, Get product, BuyEventTickets, GeeCienericBusinesst/yy Type, Order menu item fall under this type.
The disco corpus contains good representation of shorter and longer disfluent sentences across each language, increasing the complexity of cor- rections needed. Figure depicts the box plot of disfluent sentences, indicating the average sen- tence lengths of spoken uttcrances across four lan- guages. These plots summarize our analysis and motivate to test this dataset across various ML models (section 4).
Example Pay my um my phone bill for this month.
Internal Task: Sentence pairs belonging to this domain consist of utterances where the user wants the AI agent to perform task which does not involve any extra appli- cation. Examples could be sending mes- sage/email to someone, cancelling some plan, taking some notes, etc. If third-party appli- cation is used in the utterance, it is an "Ex- ternal Task"; if not, it is an "Internal Task". Domains such Get message content, Add contact, Create note, Open app, Take photo, Add item to list fall under this type.
C Domain Level Analysis of DC Results
We show the domain-wise performance of our DC models in each language in table 14. The best models can reach 100 per cent accuracy for many domain types. However, there are still some do- mains, such as Log exercise, Get Bill and Log nu- trition, where the performance varies significantly for each language. Our results show robust perfor- mance across domain as well as disfluency types (section 5.1).
Example- want to e-mail Zane this photo and cc um and cc Zach.
D Evaluating models trained on DISCO using other DC datasets
External Task: Sentence pairs belonging to this domain consist of utterances where the user wants the AI agent to perform a task with the help of third-party application. In this domain, you will find utterances where the specifies the AI agent and which appli- cation should the AI agent use complete the task. Domains such as Cancel ride, Order ride, Post message fall under this type.
Models trained DISCO outperformed test sets from other DC datasets. Table 15 shows exper- iments testing a large transformer model trained on the DISCO dataset with other open-source test sets. We ensured equal distribution across disflu- ency types while creating the test sets from DISCO, LARD (Passali et al., 2022) and Switchboard (God- frey et al., 1992) datasets. MuRIL was used the transformer for our experiments because it re- ported high F1 score across other language mod- els. Moreover, since labeled datasets are being evaluated, we could not use the Seq-GAN-BERT model as it required unlabeled data. Our experi- ments indicate that models trained on the DISCO dataset, perform better than models trained on real or synthetic data. Such experiments could not be performed for French and German because of lack of existing open source DC datasets in these lan- guages.
Example- Use WhatsApp to to send location to Jim.
We also depict the word cloud of disfluent sen- tences across the four languages. Our analysis shows the most common disfluent words across four languages. Since the Filler class occupies majority in the distribution, for each language, we see filler words like um, uh, er, and umm occupy a considerable size in the cloud for English. Simi- larly, common fillers in Hindi, German and French are the biggest in the respective word clouds (fig- ure 3).
E More examples of DC inference
Correlation analysis between original Hindi and German sentences and their respective English translations was also performed ensure that the number of outliers was minimum and the slope of points followed a natural straight line. Figure 4 de- picts the straight-line seatter plots observed.
This section shows more examples of inferences from best models across all four languages English (Table 16), Hindi (Table 17), German (Ta- ble 18) and French (Table 19). Since Hindi sen- tences are written in the Devanagari script, we pro- vide transliteration and gloss for every example

---

discussed. Strong results of our models for DC mo- tivate us test their performance for downstream MT improvement (Section 5.2)
table 20 and table 21, respectively. Some impor- tant examples where DC impacts downstream MT discussed section 5.2.1.
F
Setup for downstream DC-MT experiments and domain level results
Disfluency Correction has been studied predomi- nantly as post editing task for downstream tasks. In this section, we discuss important experiments in understanding the impact of disfluencies for down- stream machine translation. We work with the Hindi English and German English language pairs. We use the NLLB MT system (CCosta-jussá al., 2022)) for our experiments. The following steps are followed-
Disfluency Correction has been studied predom- inantly as a post-editing task for downstream prob- lems. This section discusses essential experiments in understanding the impact of disfluencies downstream machine translation. We work with the Hindi English and German English language pairs. Our experiments use the NLLB MT system ((Costa-jussà et al., 2022)). The following steps are followed
Disfluent sentences passed through the MT system to create translations TI
Automatic Disfluency Corrected sentences (using our best DC models) are passed through the MT system to create translations
Human--crreddie sentences (as provided by our annotators) is passed through the MT sys- tem to create translations T3
We denote ground truth translations as GT. TI is compared with GT to calculate BLEU DIS. Similarly, BLEU ADC is the score between T2 and GT, and BLEU HDC is the score be- tween T3 and GT.
These three BLEU scores indicate the perfor- mance of machine translation the absence of disfluency correction BLEU DIS, in the presence of automatic disfluency correction BLEU ADC and the presence of human dis- fluency correction BLEU HDC
Domain level results for Hindi-Englis and German-English sentences and changes BLEU score observed with and without DC are given in

---

Lang
Disfluent Sentence
Confusion and Resolution
En
need to construct or create new note for andy
construct note does not make sense and cre- ate could correction
The utterance does not indicate any change in speech plan, this could be semantic mistake made by non-native speaker. Output should be the same as input due to fluent sentence This example contains both repeated phrase and ce and filler um. What should be the cor- rect disfluency type label assigned? We observe that the speaker hesitates the ut- terance and ce at which point the speech plan is changed. Thus, repetition plays a more im- portant role here The second word is a filler and must be re- moved. However, should remove the first word too because tion?
want to e-mail Zane this photo and um and
Zach
Hi
रसम अोंे नही सीमाा को पका बोडस मये भोगी
lose out informa-
In this utterance, the first word a part of reparandum and hence needs to be removed too.
f अपने कमर सा कममर की साडि डामन बाहता aid
This sentence contains repetition and filler. What should we mark as disfluency type? In this example, repetition plays greater role in introducing the disfluency In this example, is the word Kick uttered correction?
De Brauche Sandalen von äh Kick.
No, Kick is the name of a shop in Germany The word pause is repeated, but is not consec- utive. What should the label be?
Pause, drück auf Pause.
In this example seems the user uttered pause but then corrected himself to give more infor- mation about what he wants to pause. Correc- tion tag seems most applicable Mets-moi à Fortnite OR jouons à Fortnite are the gramatically correct options.
Fr
mets-moi jouons
Fortnite
In this example, seems the speaker uttered the false started phrase mets-moi first and then corrected themselves. Considering it to be False Start, the first part must be removed. Should the tag be Correction Filler since ation does not mean sound/noun
Annulation ation du Uber.
ation seems like mistaken utterance and hence should be tagged as correction
Table Flagged examples data creation and our reasoning to create fluent sentences disfluency labels

---

Most Common
English
yea
atra
stttars
Facebook
uhh
cDd
Cantac
aphona
plaae
S
oh
open
mean
fum
ahh
cedn
em
sales
etraating
record
stat th
andwsday
face
S
RS
prt
discover
OPTT
let
pho
sam
con
stare
hull
month Sanssing
3115
ug
star
15
ninute
ahm
ecrning
ek
whats
uhimbiin
seat
er
well
phore
seeg
umm
yeater day
c
ST
3
अR
45
944
wa i
id
runkeeper
Ummm
Tive
Ed0e
tro
nome
log
are
snap
is
colories
Matapp
Baleko exf
want
yes
pick
need
Calliy
nom
ch
अहह
3E
M
how
Lzurch
Tithut
beok
mill
3R मह
F
U
mall
ton
ah
name
anipren
yhm
yester
last
cate
CH SAT
3 3TT मललय 5
R
activity
actualll
balled
completed
guess
orden
Linsez
shon
instore
pie Wuls
Uuadday
wet
ytp
Common
Disflent Words German
Mext Common Diefuent Words French
U
a
Johm uhm
ann
ein eine
Betzt
pause
blut
face
insta
vor
stup
VIBeD
uh
nich
beenden
uber
das
Vie
für
yn
du
ehh
wark
sagen
hoch
mh lyft lass
stra
lan
lan
non
mmmh
vie
de
COmmETT
re
instajai
Ffinun
ehn
OUL
euuuuh
CORN
Un
commander
CUMTAR
ffiche... diina
por
pouF F
CO
unh
ale
öwsstarte..
zeig
nach
lauf
nein
Sääh
der
kowwer
wwillmachhhee
mit
eut
relance
comandee
mi
enregister
fit
euuuh
t
loppli
no
eunue
train
ich
neue
@mmm
fort
meine
ummannull
cour
tu
hmmh
doch
ach
yogé
brauche
pau
dem
nache mehr
euhhh
unger
riicc
Deu
pe
donne
hmmm hhaney
uhnn
niwn
tach
rich
app
dune
ahhh
brauch
app
e
ahh stopp
hmm
web
wetriinierg
tar
ma
fy
an
tax
ihh
que
U/prgg
bitte
auf
bestell
mache
die
bwm
mmmm
pare
train
mmm
Perter
will
envoyer
mmmmh
he
annuter
Lhee
mes
paiser
heu
etsor
meine
aäähm kannst
ehm
genau
jöhh
meinen
austere
emm
eure
machen
emmmm
nanzu
wuhn
kurhe
beatinee
euhh
continuer
chrehmin
arrice
bo
a jajaaaojttt ajoute
joue
triepes
ouvr
multi
stratter
e
peau
pau
mont
poogie
Ruen
Figure 3: Word cloud of disfluent sentences across each language in the DISCO corpus, showcasing the most
common disfluent words observed in spoken utterances
Sentence Length Correaation
English
Length Comelation
English
17%
15.0
Orgins Semeraa
200
75
35
2.5
Hind Sentance
Gerean
Figure 4: Plotting the correlation between disfluent sentences and their English translations. These graphs indicate that the number of words in any English translation of Hindi or German sentences can be estimated using straight line slope as depicted with minimum outlier cases.

---

Sentence Length Distribution for all languages
30
25
0000000
0000
Number of Words
20
15
10
5
English
German
French
Languages
Figure 5: Box plot of disfluent sentence lengths across all languages in DISCO corpus

---

Domain Type
English
Hindi
German
French
Send digital object
222
116
116
121
Get health stats
258
105
140
95
Get message content
191
68
69
100
Add contact
286
143
106
118
Create note
68
62
80
45
Check order status
274
150
90
72
Get bill
255
131
82
68
Get security price
238
117
85
68
Open app
238
156
185
226
Pay bill
251
128
109
105
Get product
218
94
116
98
Other
223
72
89
62
Post message
256
134
178
169
Record video
25
129
152
154
Log exercise
228
129
128
73
Log nutrition
248
88
98
87
Take photo
0
108
140
150
Cancel ride
0
191
109
170
Order ride
138
86
148
BuyEvennTickets
98
118
85
Play game
0
115
133
151
GetGenerieBusinessIIT Type
0
103
101
38
Start exercise
0
164
151
133
Stop exercise
0
172
170
164
Pause exercise
0
129
157
100
Resume exercise
0
140
108
165
Order menu item
0
0
0
25
Add item list
Table 13: Language wise distribution of domain of disfluent sentences DISCO corpus

---

Domain Type
English
Hindi
German
French
Send digital object
97.37
97.56
100.0
100.0
Get health stats
98.55
92.68
100.0
100.0
Get message content
98.36
85.71
90.0
100.0
Add contact
97.56
86.49
100.0
100.0
Create note
89.66
100.00
82.35
100.0
Check order status
98.08
93.88
95.65
100.0
Get bill
100.00
57.78
100.0
92.31
Get security price
100.00
91.89
84.21
90.91
Open app
100.00
97.87
92.31
100.0
Pay bill
100.00
92.31
100.0
94.44
Get product
100.00
88.00
100.0
96.97
Other
100.00
100.00
100.0
100.0
Post message
97.30
93.88
100.0
100.0
Record video
100.00
100.00
96.3
95.83
Log exercise
94.12
92.00
100.0
92.31
Log nutrition
98.31
66.67
100.0
100.0
Take photo
100.00
100.0
96.15
Cancel ride
98.51
90.0
100.0
Order ride
94.87
98.04
100.0
BuyEventTickets
92.68
100.0
100.0
Play game
94.44
94.92
97.67
GelGennericBBsiiessTTyy
94.12
100.0
75.0
Start exercise
87.50
97.96
96.55
Stop exercise
94.74
100.0
98.11
Pause exercise
85.11
93.33
85.71
Resume exercise
94.12
100.0
90.0
Order menu item
100.0
Table 14: Fl scores for disfluency correction for every domain type in each language
Language
Test Dataset
Model Name
Test F1
English
DISCO
MuRIL. finetuned on DISCO
96.65
MuRIL finetuned on Switchboard
86.97
MuRIL finetuned on LARD
89.33
Switchboard
MuRIL finetuned on DISCO
95.67
MuRIL finetuned Switchboard
88.44
MuRIL finetuned LARD
89.33
LARD
MuRII finetuned on DISCO
96.21
MuRIL finetuned on Switchboard
91.23
MuRIL finetumed on LARD
97.30
Hindi
DISCO
MuRIL finetuned on DISCO
94.29
MuRIL finetuned on Kundu et al. (2022)
81.16
(Kundu et al., 2022)
MuRIL finetuned on DISCO
87.16
MuRIL finetuned on Kundu et al. (2022)
84.81
Table 15: Test F1 scores of transformer based model trained on the DISCO dataset compared other open-source datasets. For LARD, our model performs comparatively even in out-of-domain synthetic test sets.

---

Lang
Type
Disfluent Sentence
Predicted Fluent tence
Sen-
Comments
En
F,R
ask chase how much do uh how much my visa bill is put 203 wait 2086683553 into my contacts text sal at 4556329902 ac- tually 4556329092
ask chase how much my visa bill is put 2086683553 into my contacts text sal at 4556329092
Able to remove more than one type of disfluency
C
numerical disfluency iden- tification
C
FS
कलीका पाīęड नरा मतलय ? सादकर मड सेन पाोन के सयडीड डील बमा
साडनर मडे पर सील पाोन के सहडे डील बया ?
Utterance had complete change in content, de- tected correctly
C
lets record on fitbit that did 10 pushups wait make that 20 pushups this morn- ing
lets record on fitbit that did make that 20 pushups this morning
Incomplete correction
FL
need to construct cre-
need to create note for andy
Tradeoff between correc- tion and content preserva- tion
new note for andy
ER
make my status on face- book um make it anyone down for a movie
make my status on face- book make anyone down for a movie add the number 5551987 to my husbands to his work contacts list what was my water in in- take last month
Issues when more than one type of disfluency is present
F.C
add the number 5551987 1987 to my husbands to his umm work contacts list what was my water in in intake last month
example where normal syntactic structure of dis- fluencies is not followed Model mistakes correc- tion as repetition
C
Table 16: Inference examples from English DC models; English

---

Lang
Type
Disfluent Sentence
cancel do it tomorrow Transliteration Gloss
Predicted Fluent Sen- tence
Comments
Hi
पिके Fहा का नरी 3
Pichhale mahīne did
last month
पिकर महो
Able to remove
my
की नरी का
multiple occur-
कार पिातनी
की
meri um- ummm
cी पिातनी
rences of fillers
$ mma kär car did उंअ डरमअाह पकाापा ?
डरमअाड कपायपा t
kt kitanī howmany aa emi arrears is akinator Of level start do two do
aa Temaäi
bakāyā hai
परितीटर का लेल 4 y दो कर शर
Ekinetar kä
पपितीरर का Solves ambiguity to maintain gram- mar
leval shurū kar do kar do shurā
लेलल q
कर बी
Two start cab book No बा करसल
C
काब 9क नहा करित करर टी काल शाम बी
Kaib buk nahian ka-
Chain in thought process detected & corrected
कार टी कhरी
iansil kar do
शाम बी पिररीी के बिद अोला
शितरी के शिद अोला
kal shäm ki evening did
dilli ke lie olä para
delhi of for ola on black friday My Meaning Is cyber Monday
पर
FS
कोक Uniss
Blaik fräide merä mata- lab hai sāibar
Utterance had com- plete change in con- tent, detected cor- कया àे rectly
साककर मड पर रोल पहो के
मेश मसलयप è साडकर मडड पर रोल पगोन के सकडेв डील कपा ?
सकडेड डील
mande par sel Don ke sar-
Cell phone
of the best deal what Is
vashreşhtha dil kyä hai डोमननंग Daīminoj
F.
Domino's डोमनोयग पकक अगम se ek am- दो मोoडपम mma do
removes filler but does not make correction
Model
From One mmmm Two
पकक
दी
मीडडपम
पकरशा सीडश mīdiyam सपीर कोन eksträ chij extra Thing
medium
प सीश सीीट
कोन भ
पपगा aR अहउा डबिद काड से ली
svīt ka
sweet
com
पिगा
pijajaa mere
pizza
My
डेिि काड से
aaa debit
AAA
debit
ली
kärda se
card From
Take
R
पमम पर आारा पंटा बाया संटा पारर रीिप श कर टी अभभी ले अभभी
fon par
Phone
On
माोन पर बाе Model removes पनरर
ädhä ghantä
half
Bell
शंटा
repetition but also treats the following phrase as a possible disfluent utterance: अभभी 7े अपी
ädhä ghantä
half Bell fast
रपिा शाह करर
āşta rani-
running Start
टी अfभी
anga shurū
do Two Now
kar do abhi
ke abhi Of Now
Table 17: Inference examples from Hindi DC models; Hindi

---

Lang Type
Disfluent
Sen-
Predicted Sentence
Fluent Comments
tence
De
F
ich muss öhh
ich muss um 1830
Filler öhh correctly iden- tified and corrected. um not a disfluent word in German and the model correctly recognizes this. vom is repeated with the filler emmm in between which causes ambiguity regarding the phone bill (telefonrechunng). Model correctly identifies and re- moves the disfluent utter- ances Speaker starts with the phrase mache ähm but changes speech plan to ut- ter the latter fluent phrase. Model correctly removes false started phrase
1830 am flughafen
am flughafen sein
sein
F,R
die telefonrech- die telefonrech- nung vom emmm vom letzten monat über paypal bezahlen
nung vom letzten monat über paypal bezahlen
FS
mache ähm kannst du ein video in hd machen
kannst du ein video in hd machen
hey fedex zeige mir den status von den status der bestellung mit der nummer 111222abc
hey fedex zeige mir den status von sta-
den status is repeated in the original phrase but the model only removes one of the repeated words, out- put remains disfluent. In this case, repetitions are not consecutive and thus relatively difficult for the model to correct example where normal syntactic structure of dis- fluencies is not followed
tus der bestellung mit der nummer 11 111222abc
R,C
übung für auf
strava auf
übung für
strava übung auf pause
Table 18: Inference examples from German DC models; De - German

---

Lang
Type
Disfluent
Sen- Predicted Fluent Comments
tence
Sentence
F
Ouvrir euh
ouvrir lapplication Filler euh correctly de- tected and removed by model
l'application
facebook sur mon iphone
Facebook sur mon Iphone. Envoyez mon mes- sage euuh mon mes- sage audio.
F.R
envoyez mon mes-
User repeats the phrase mon message but the sec- ond time they mention that it is audio message. Hence the earlier repeated phrase is removed along with the filler word euuh User asks for discounts and then corrects by ask- ing for discounts informa- tion. Model correctly de- tects this
sage audio
Trouver remises des infos sur les remises sur les remises.
des trouver des infos
FS Annule, cuh, tu peux annuler la peux annuler la course course.
User starts phrase can- cel, then corrects to ask cancel the course. Model identifies the false started phrase and removes relancer is corrected to relance but the model doesnt detect this
C J'aimerais relancer relance mon en- trainement HIIT. trainement hiit Enregistrer une enregistrer une Model incorrectly thinks hd is disfluent term
jaimerais relancer relance mon en-
F
vidéo HD avec
vidéo avec insta-
hmm Instagram.
gram
Table 19: Inference examples from French DC models; fr French

---

Domain Type
MT without DC
MT with ADC
MT with HDC
Send digital object
49.17
56.20
53.67
Get health stats
28.05
30.77
34.69
Get message content
27.62
28.72
26.42
Add contact
32.29
48.34
41.81
Create note
42.18
47.63
47.63
Check order status
44.03
45,50
46.96
Get bill
44.80
57.29
47.33
Get security price
39.91
51.98
52.50
Open app
43.40
50.08
51.65
Pay bill
41.38
49.83
50.16
Get product
31.37
35.25
33.37
Other
22.39
22.58
22.58
Post message
42.46
48.56
48.66
Record video
50.34
62.23
62.23
Log exercise
30.92
29.37
29.02
Log nutrition
26.21
31.34
33.08
Take photo
44.13
42.38
42.38
Cancel ride
36.29
42.36
45.14
Order ride
33.77
41.93
40.71
BuyEven(Tickets
27.96
23,43
24.67
Play game
30.04
42.11
47.67
GeeGGnnericBusiiesstTyyy
32.92
37.16
42.32
Start exercise
26.91
48.18
49.56
Stop exercise
31.89
38.59
43.05
Pause exereise
32.91
39.95
47.76
Resume exercise
33.67
36.96
36.96
Table 20: Efffect of disfluency correction on downstream machine translation across different domain sentences for Hindi English pair

---

Domain Type
MT without DC
MT with ADC
MT with HDC
Send digital object
56.01
87.20
87.20
Get health stats
41.56
53.79
53.79
Get message content
15.59
34.62
34.62
Add contact
33.94
41.86
41.86
Create note
35.83
52.92
52.56
Check order status
49.79
59.07
61.56
Get bill
49.52
58.31
58.31
Get security price
34.71
37,99
40.11
Open app
49.71
61.38
61.38
Pay bill
44.70
73.85
73.85
Get product
31.64
33.25
33.25
Other
36.75
44.12
44.12
Post message
46.63
72.00
72.00
Record video
38.29
46.65
46.65
Log exercise
30.76
40.70
40.70
Log nutrition
41.90
37.42
37.42
Take photo
41.63
38.45
38.45
Cancel ride
48.23
54.91
54.91
Order ride
24.55
42,42
42,42
BuyEven(Tickets
44.55
44.31
44.31
Play game
36.77
59.36
59.36
GeeGGenericBuuiinsstTyyy
43.64
51.06
51.06
Start exercise
45.30
48.37
48.37
Stop exercise
26.94
21.49
21.49
Pause exereise
31.57
36.14
38.33
Resume exercise
51.39
46.89
46.89
Table 21: Efffect of disfluency correction on downstream machine translation across different domain sentences for German English pair


Reviews:
[{"review_id": "HsLAxZxVix", "reviewer": "Reviewer_J4JB", "summary": "", "questions": "Question A:  Are you able to perform some experiments like what I suggested in reasons to reject?  I think some things like--showing how your dataset improves model performance on other DC test sets, how other datasets don't perform as well on your (or their) DC test sets, and how other DC models from existing datasets don't help downstream MT as much as models trained with your dataset do--would greatly strengthen the claim that your dataset is useful.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "_Question A: Are you able to perform some experiments like what I suggested in reasons to reject? I think some things like--showing how your dataset improves model performance on other DC test sets, how other datasets don't perform as well on your (or their) DC test sets, and how other DC models from existing datasets don't help downstream MT as much as models trained with your dataset do--would greatly strengthen the claim that your dataset is useful._\n\n**Author Response:**\n\nThank you for your suggestion. \n\nAfter testing models finetuned on various datasets to analyse the robustness of models using the DISCO dataset, we present results of a few additional experiments below. \n\nWe get the following results for English DC - \n\n| Test Dataset | Model name | Test F1 score |\n|----------|----------|----------|\n| DISCO  | MuRIL finetuned on DISCO (English only)  | 96.65 | \n|   | MuRIL finetuned on Switchboard  | 86.97  | \n|  | MuRIL finetuned on LARD  | 89.33  | \n| Switchboard  | MuRIL finetuned on DISCO (English only) | 95.67 | \n| | MuRIL finetuned on Switchboard | 88.44 |\n| | MuRIL finetuned on LARD | 89.33 |\n| LARD  | MuRIL finetuned on DISCO (English only) | 96.21 | \n| | MuRIL finetuned on Switchboard | 91.23 |\n| | MuRIL finetuned on LARD | 97.30 |\n\nSimilarly we get the following results for Hindi DC - \n\n| Test Dataset | Model name | Test F1 score |\n|----------|----------|----------|\n| DISCO  | MuRIL finetuned on DISCO (Hindi only) | 94.29 | \n|   | MuRIL finetuned on Kundu et al. (2022)  | 81.16  | \n| Kundu et al. (2022) (only 100 sentences)| MuRIL finetuned on DISCO (Hindi only) | 87.16 | \n|   | MuRIL finetuned on Kundu et al. (2022)  | 84.81 | \n\n\nThus we observe that in all cases, models trained on the DISCO dataset, perform better than models trained on real or synthetic data. For LARD, our model performs comparatively even in out of domain synthetic test sets. Such experiments could not be performed for French and German because of lack of other open source DC datasets in these languages.\n\nWe were also able to compare the performance of other DC systems on the Hindi - English DISCO MT improvement task when used with a state-of-the-art MT system (NLLB). The following BLEU scores were obtained when using the different models as described above: \n\n| Model name | BLEU score |\n|----------|----------|\n| MuRIL finetuned on DISCO (Hindi only)   | 42.63 | \n| MuRIL finetuned on Kundu et al. (2022)  | 38.97  | \n\nThese results additionally strengthen the validity of our dataset as a good resource to train DC systems for downstream MT improvement in conversational settings. \n\n**Additional Response:**\n\nWe would like to highlight that our proposed dataset can be used to train DC systems that can be used as a preprocessing step before any downstream processing such as machine translation, intent classification, etc. Since existing SOTA MT systems are not trained on disfluent text, such a pre-processing step can improve downstream performance as illustrated in section 5.2. Additionally, the data we created for disfluent Hindi - English and German - English translation tasks can be used to train better performing conversational MT systems. \n\n**Final Comments**\n\nThe authors sincerely thank the reviewer for their time to assess and evaluate our submission.\n\nWe apologize for the lack of clarification of these above points and would be happy to include these explanations in the revised version of this paper in appropriate tables and sections.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The dataset seems useful and well-designed with respect to the collection protocol.\n\nThe authors are able to clearly show an increase in correction and downstream MT performance using their presented data.\n\nThere are some flaws discussed below, but considering the inclusion of under-represented languages in DC, it's pretty clear that the contribution is a good one.", "reasons_to_reject": "The main thing that determines the strength of a dataset paper is how well its usefulness is demonstrated.  The authors do this by performing disfluency correction evaluation as well as downstream MT evaluation.  They show that they are able to achieve high scores, varying by language/type of disfluency.  They also show that using this dataset improves performance of MT.\n\nThe main issue here is that the authors have demonstrated well how 'disfluency correction' is useful as opposed to how *their* disfluency correction data is useful.  We see that better models perform better on the dataset for example.  This helps to validate the dataset as sound, but there are no experiments to compare to how using data that is already available performs.\n\nSimilarly for MT, they show that both human and automatic (using their data) disfluency correction improve MT, but not how *their* disfluency correction contributions improve MT.  They mention this as a limitation and future work, but I think it is a fairly basic necessity for a strong dataset paper.", "questions_to_authors": "Question A:  Are you able to perform some experiments like what I suggested in reasons to reject?  I think some things like--showing how your dataset improves model performance on other DC test sets, how other datasets don't perform as well on your (or their) DC test sets, and how other DC models from existing datasets don't help downstream MT as much as models trained with your dataset do--would greatly strengthen the claim that your dataset is useful.", "paper_topic_and_main_contributions": "This paper's contribution is a new disfluency correction dataset.  The dataset includes human-annotated examples from 4 languages (de,en,fr,hi) as well as translations to en for two of them (de-en,hi-en).  The authors claim that the dataset is useful with experiments showing improved disfluency correction capabilities as well as its impact on downstream NLP task performance.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "o5So25WZHf", "reviewer": "Reviewer_z3XW", "summary": "", "questions": "1. It is unclear why the four languages considered for the paper were chosen. There may be other low-resource Indo-European languages that require attention, so why these four?\n\n2. It is unclear from the paper that how the helper datasets were used for few-shot learning-based experiments on DISCO.\n\n3. Table 5 shows that the performance of Seq-GAN-BERT varies significantly across languages, and the authors' claim that this model performs well in low-resource settings via adversarial training is not adequately reflected. The lack of explanation on this table exacerbated the situation. \n \n4. Despite having a small number of filler type data samples (as shown in Figure 2), the best DC model performs exceptionally well for this type of disfluency across all languages (as shown in Table 6). What is the reason for this? At the very least, attempting to address these issues makes the paper far more interesting. ", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "_Question 1: It is unclear why the four languages considered for the paper were chosen. There may be other low-resource Indo-European languages that require attention, so why these four?_\n\n**Author Response**\n\nWe derive our source sentences from the PRESTO dataset (Goel et al. (2023)) which contains the following languages - English, Hindi, German, French, Spanish and Japanese. However, since we only had qualified annotators in English, Hindi, German and French, we chose these four languages. By doing so, we covered two widely spoken languages (German and French) which did not have any open source DC datasets, having different linguistic properties to provide a resource to study various types of disfluency phenomenon and its properties. \n\n_Question 2: It is unclear from the paper that how the helper datasets were used for few-shot learning-based experiments on DISCO._\n\n**Author Response**\n\nWe used unlabeled disfluent/fluent sentences from the mentioned helper datasets for few shot learning (lines 298-299). The Seq-GAN-BERT model (Bhat et al. (2023)) has been used before to train DC models in extremely low resource settings using few shot learning to achieve state-of-the-art performance and thus we wanted to evaluate its performance on our DISCO dataset. \n\n_Question 3: Table 5 shows that the performance of Seq-GAN-BERT varies significantly across languages, and the authors' claim that this model performs well in low-resource settings via adversarial training is not adequately reflected. The lack of explanation on this table exacerbated the situation._\n\n**Author Response**\n\nThe Seq-GAN-BERT model has shown to perform well in extremely low resource settings (where we only have around 300-400 labeled sentences). However when the number of sentences increases, we observed that the performance saturates and does not reach very high numbers, as compared to other transformer architectures. Moreover, Seq-GAN-BERT was only tried before for English and 2 other Indian languages (Hindi and Marathi) and this paper is the first work for testing it on European languages. Since Indian languages have significant differences from the chosen European languages, that could be a reason why Seq-GAN-BERT underperforms. However, general numbers for all models are in the range of >85 F1 score, which indicates that our dataset can cater to a wide variety of sequence tagging algorithms.\n\n_Question 4: Despite having a small number of filler type data samples (as shown in Figure 2), the best DC model performs exceptionally well for this type of disfluency across all languages (as shown in Table 6). What is the reason for this? At the very least, attempting to address these issues makes the paper far more interesting._\n\n**Author Response**\n\nPrevious work has shown that DC models generally perform well for filler disfluencies (Kundu et al. (2022), Passali et al. (2022)) since they only contain simpler syllables like “um”, “err”, “agh”, which are easily identified by transformers as erroneous utterances. Moreover there are many sentences in all languages that have more than one type of disfluency present, with fillers being the most common additional disfluency (Appendix B: figure 3).\n\n_Missing References: The authors' claim that CRFs and RNN-based techniques require less labelled data and are ideal for low-resource domain-specific training is not supported by appropriate references._\n\n**Author Response**\n\nThe performance of CRF and RNN based systems in low resource settings is studied in Simpson et al. (2020) where they use LSTM and CRF based sequence taggers for named entity recognition in low resource setups. We will include this reference in the revised version of our paper.\n\n_Typos Grammar Style And Presentation Improvements:_\n\n_Many places in the paper, for example, Tables 2 and 7, contain non-English languages without translation, which may be difficult for non-native speakers. Appropriate translations might be provided in these cases._\n\n_There are a lot of figures and tables in the appendix section, but no reference text. Their presentation is not only unappealing, but as there is no explanation for these data, their utility to readers may be deemed unnecessary._\n\n**Author Response**\n\nWe apologize for the lack of English translations in table 2 and table 7, which were excluded due to space constraints. Alternatively we tried to mention appropriate examples in all languages so that the data can be understood by a wide range of readers. Due to space constraints we could not include English translations in the paper, but we will be happy to add English translations of all the examples in the revised version of our paper. \n\nWe apologize for the missing explanations and captions in the appendix. We provided a forward and back reference to the main sections and appendix, but we will improve the readability of the appendix by adding more explanations in key sections and captions wherever necessary in the revised version of this paper.\n\n**Final Comments**\n\nThe authors sincerely thank the reviewer for their time to assess and evaluate our submission.\n\nWe apologize for the lack of clarification of these above points and would be happy to include these explanations in the revised version of this paper", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This well-written paper provides a high-quality dataset on Disfluency Correction (DC) on four important languages, including French and German, which do not have a DC corpus other than the one proposed by the authors, as they claim. This dataset can be a good resource for improving language understanding and translation task. ", "reasons_to_reject": "1. Rather than presenting quantitative statistics in tables, the paper's results section is not well-organized and offers few insights into the impact of disfluency correction. More in-depth discussion of the performance improvements brought about by DC, as well as future directions, would be beneficial to the reader. \n\n2. Because there are already a lot of works done in English for DC, the authors chose English as part of their corpus again, and the motivation for this is unclear. The reason for the languages considered for the work is not stated explicitly. \n\n3. The annotation process only used one annotator per sample, resulting in a significant lack of annotation validity. Moreover, the paper does not specify whether the annotation was done by native speakers.", "questions_to_authors": "1. It is unclear why the four languages considered for the paper were chosen. There may be other low-resource Indo-European languages that require attention, so why these four?\n\n2. It is unclear from the paper that how the helper datasets were used for few-shot learning-based experiments on DISCO.\n\n3. Table 5 shows that the performance of Seq-GAN-BERT varies significantly across languages, and the authors' claim that this model performs well in low-resource settings via adversarial training is not adequately reflected. The lack of explanation on this table exacerbated the situation. \n \n4. Despite having a small number of filler type data samples (as shown in Figure 2), the best DC model performs exceptionally well for this type of disfluency across all languages (as shown in Table 6). What is the reason for this? At the very least, attempting to address these issues makes the paper far more interesting. ", "paper_topic_and_main_contributions": "This paper presents a human-annotated Disfluency Correction (DC) dataset for Indo-European languages, including English, Hindi, German, and French, as well as some benchmark models. In this paper, the authors deal with four types of disfluencies: filler, repetition, correction, and false-start. At the end of the paper, some analysis of the benefits of DC is also discussed. ", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "1. The authors' claim that CRFs and RNN-based techniques require less labelled data and are ideal for low-resource domain-specific training is not supported by appropriate references.\n\n"}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. Many places in the paper, for example, Tables 2 and 7, contain non-English languages without translation, which may be difficult for non-native speakers. Appropriate translations might be provided in these cases. \n\n2. There are a lot of figures and tables in the appendix section, but no reference text. Their presentation is not only unappealing, but as there is no explanation for these data, their utility to readers may be deemed unnecessary. "}}, {"review_id": "fiNds6lm3H", "reviewer": "Reviewer_HAs1", "summary": "", "questions": "Question A:  How much of the data for each language has paired fluent sentences? \n\nQuestion B:  It is stated that labels are predicted for each word. Model vocabularies are not mentioned in the paper but the pretrained models mentioned operate at the subword level.  Is prediction done at the subword level? How are situations where different labels are predicted for different subwords handled, a majority vote?", "limitations": "", "rating": 3, "confidence": 2, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "_Question A: How much of the data for each language has paired fluent sentences?_\n\n**Author Response**\n\nThe data mentioned in table 3 specifies the amount of parallel data for every language. Thus our corpus contains 3479 disfluent-fluent sentence pairs in English, 3180 disfluent-fluent sentence pairs in Hindi, 3096 disfluent-fluent sentence pairs in German and 3005 disfluent-fluent sentence pairs in French. We will specify this clearly in the caption of table 3.\n\n_Question B: It is stated that labels are predicted for each word. Model vocabularies are not mentioned in the paper but the pretrained models mentioned operate at the subword level. Is prediction done at the subword level? How are situations where different labels are predicted for different subwords handled, a majority vote?_\n\n**Author Response**\n\nDuring the tokenization step, the tokenizer assigns an ID to every sub word. The first subword is marked to be labelled whereas the other subwords from the same word are ID'ed as -100 and hence ignored for prediction. Thus, the prediction of a word to be disfluent/fluent is the prediction of the first subword to be disfluent/fluent. We will include this clarification in the revised version of the paper appropriately. \n\n_Missing References:\nhttps://arxiv.org/abs/1811.03189 [Salesky et al. (2018)] also collects paired fluent English sentences for originally disfluent English and should likely be cited_\n\n_This work is not the first to create labeled corpora for disfluency labeling / removal in German or French though this is stated several times in the paper: see https://aclanthology.org/L14-1277/ [Cho et al. (2014)], https://arxiv.org/abs/1802.02926 [Christodoulides et al (2018)], https://link.springer.com/chapter/10.1007/978-3-540-87391-4_35 [Bove et al. (2008)], and others cited in each and citing these_\n\n**Author Response**\n\nWe apologize for some of the missed references and thank the reviewer for pointing them out. We will be mentioning Salesky et al. (2018) in section 2 appropriately. However, there are some points we wanted to highlight regarding our assertion of being the first to create and share DC datasets in German and French. Although Cho et al. (2014) annotated the KIT lecture corpus for disfluencies in German, they were never shared the dataset open source. Christodoulides et al (2018) created DisMo, a tool to detect disfluencies but they use language clues to do it, and do not create a tagged disfluency dataset. Similarly, Bove et al. (2008) created a system to predict disfluencies using part of speech tags, without creating any dataset. To avoid confusion, we will mention that our work is the first “open-sourced” dataset for DC in the given languages. These past works are important to be noted and hence will be mentioned appropriately in section 2 of the revised version of this paper. \n\n_Typos Grammar Style And Presentation Improvements:\nIt is not clear to me how much of the data has paired fluent sentences; for example Table 3 suggests that the data is paired with disfluent-fluent sentences and Sec 3.3 says annotators were asked to create fluent sentences corresponding to disfluent utterances (as in, for each disfluent sentence there is a fluent rewrite) whereas Figure 2 shows that there fewer than 100 fluent sentences for two of the languages and fewer than 400 for the other two though there are 3k+ for each of the four languages._\n\n_Calling the task of tagging / classification 'disfluency correction' / DC rather than disfluency classification, labeling, or tagging is a little unclear in for example Sec 5.1 which it is a strict classification task and a slight departure from past work; consider rephrasing_\n\n**Author Response**\n\nFigure 2 demonstrates the distribution of our dataset across the types of sentences - Filler, Repetition, Correction, False Start and Fluent Sentences. Here “fluent sentences” are source sentences that did not have any disfluencies, and hence were not annotated for disfluencies. Hence for every source sentence in our corpus (which might or might not have disfluencies), there is a corresponding fluent sentence. \n\nIn previous literature, DC has been referred to when doing sequence tagging tasks and hence we follow the same norms. However, we will rephrase that our work is a token classification based DC in the revised version wherever appropriate. \n\n**Final Comments**\n\nThe authors sincerely thank the reviewer for their time to assess and evaluate our submission.\n\nWe apologize for the lack of clarification of these above points and would be happy to include these explanations in the revised version of this paper\n\nIn view of the above answers, we request the esteemed reviewers to increase the score.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- New annotated datasets for disfluency tagging and potential removal in four languages (English, German, French, Hindi) \n- Comparison of several models for disfluency tagging in four languages, and demonstration that the removal of disfluencies tagged using such models leads to noticeable improvements in downstream MT performance for Hindi and German to English using NLLB (which was not trained on disfluent data)\n- Examples and qualitative analysis of trends", "reasons_to_reject": "- The composition of the dataset is not clear (how many disfluent sentences have paired fluent sentences?)\n- Limited comparison to past approaches and awareness of past work for French/German \n- F1 appears to be reported for the classification task of fluent/disfluent, as opposed to for the different class labels, which is the slightly more challenging task typical of past work", "questions_to_authors": "Question A:  How much of the data for each language has paired fluent sentences? \n\nQuestion B:  It is stated that labels are predicted for each word. Model vocabularies are not mentioned in the paper but the pretrained models mentioned operate at the subword level.  Is prediction done at the subword level? How are situations where different labels are predicted for different subwords handled, a majority vote?", "paper_topic_and_main_contributions": "This paper constructs a disfluency tagging dataset for four languages (English, Hindi, German, and French) using ~3k sentences for each language selected from Goel et al. (2023), a dataset collecting human-AI agent dialogues for a variety of different domains.  \nOne annotator per language tagged four types of disfluences (fillers, repetitions, corrections, false starts) and created fluent paired sentences for each sentence (?).  \nSeveral approaches are compared for disfluency classification (prediction of a disfluent or fluent label for each word) for all four languages, including CRFs and RNNs using features from XLM-R; three different multilingual encoders with classification heads attached; and Seq-GAN-BERT.  \nIt is shown that removing disfluent-tagged words improves downstream MT performance with the NLLB model for German and Hindi (not shown for the other two languages pairs).", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "[https://arxiv.org/abs/1811.03189](https://arxiv.org/abs/1811.03189) also collects paired fluent English sentences for originally disfluent English and should likely be cited \n\nThis work is *not* the first to create labeled corpora for disfluency labeling / removal in German or French though this is stated several times in the paper: see [https://aclanthology.org/L14-1277/](https://aclanthology.org/L14-1277/), [https://arxiv.org/abs/1802.02926](https://arxiv.org/abs/1802.02926), [https://link.springer.com/chapter/10.1007/978-3-540-87391-4_35](https://link.springer.com/chapter/10.1007/978-3-540-87391-4_35), and others cited in each and citing these", "Typos_Grammar_Style_and_Presentation_Improvements": "It is not clear to me how much of the data has paired fluent sentences; for example Table 3 suggests that the data is paired with disfluent-fluent sentences and Sec 3.3 says annotators were asked to create fluent sentences corresponding to disfluent utterances (as in, for each disfluent sentence there is a fluent rewrite) whereas Figure 2 shows that there fewer than 100 fluent sentences for two of the languages and fewer than 400 for the other two though there are 3k+ for each of the four languages. \n\nCalling the task of tagging / classification 'disfluency correction' / DC rather than disfluency classification, labeling, or tagging is a little unclear in for example Sec 5.1 which it is a strict classification task and a slight departure from past work; consider rephrasing"}}, {"review_id": "qnn6APDe88", "reviewer": "Reviewer_3wc9", "summary": "", "questions": "- L166: How do you select sentences? Random, by hand, or using some heuristic?\n- Table 3: There appears to be a very high proportion of disfluent words in the dataset. Combined with Figure 2 it appears that the dataset contains very few negative samples (fluent/non disfluent utterances). Doesn't that mean that the dataset heavily biases systems to predict words as disfluent? Could this possibly explain the outliers in Table 6?\n- L196: You first write that there is only one annotator per language and then say that the *authors* take a majority vote to choose which samples to remove. Do all authors speak the four languages? How does that work?\n- Figure 2: The figure displays skewed distributions across disfluency types, with certain types being barely present. How does this impact the usefulness of the dataset? Wouldn't it make sense to have a more balanced dataset?\n- Section 4.2.3: What's the reason for including the Seq-GAN-BERT model? The section mentions that it might be useful in low-resource settings, however, it appears to be worse than most other systems and best when applied to English.\n- Section 5.2: It's unclear what this experiment tells us. It appears straightforward that if we evaluate an MT system that is trained on clean source text, the results should be worse if evaluated on disfluent source input. Wouldn't there be more informative downstream tasks to evaluate on? E.g. intent classification.\n- ", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "_Question 1: L166: How do you select sentences? Random, by hand, or using some heuristic?_\n\n**Author Response**\n\nFor English and Hindi, we took all the sentences from the PRESTO dataset (Goel et al. (2023)). For German and French since we had budget constraints, we had to shortlist around 3000 sentences in each language. We did so by trying to maintain a constant distribution of sentences based on the length of the sentence, trying to cover shorter sentences (with easier disfluencies) to longer sentences (with more difficult disfluencies). We will include graphs depicting the sentence length distribution in the revised version of the paper. \n\n_Question 2: Table 3: There appears to be a very high proportion of disfluent words in the dataset. Combined with Figure 2 it appears that the dataset contains very few negative samples (fluent/non disfluent utterances). Doesn't that mean that the dataset heavily biases systems to predict words as disfluent? Could this possibly explain the outliers in Table 6?_\n\n**Author Response**\n\nYes, and we have mentioned this conclusion in section 5.1 (line 345). Due to the lack of fluent sentences in our source corpus (PRESTO), the dataset we create has fewer fluent sentences. This, along with lesser number of fluent sentences in the test set, is a major reason for the skewed results in table 6. Fluent sentences are more readily available from sources such as newspapers and online websites like Wikipedia and thus can be included in appropriate experiments. Since our source sentences were from a very specific domain of human-AI interactions, we did not add fluent sentences from other datasets. \n\n_Question 3: L196: You first write that there is only one annotator per language and then say that the authors take a majority vote to choose which samples to remove. Do all authors speak the four languages? How does that work?_\n\n**Author Response**\n\nWe apologize for any confusion caused because of our phrasing and we will clarify these doubts in the revised version of the paper. Since sequence tagging based DC is a relatively straightforward task, the authors used translation tools and subject knowledge to solve ambiguity. More information regarding how we deal with ambiguous cases can be found in Appendix A. \n\n_Question 4: Figure 2: The figure displays skewed distributions across disfluency types, with certain types being barely present. How does this impact the usefulness of the dataset? Wouldn't it make sense to have a more balanced dataset?_\n\n**Author Response**\n\nSkewed datasets are very common in disfluency datasets. For example, the most popular DC dataset- the Switchboard corpus has a skewed distribution towards more common disfluency types such as Fillers and Repetitions. This is attributed to the human tendency to utter these more than the other types. Although our dataset is also skewed towards these types, we also annotate a good amount of sentences in more complicated disfluencies such as false starts and corrections. We hypothesize that such contributions will be able to build DC systems that perform better across all types of disfluencies, and this is supported by our results in table 6.\n\n_Question 5: Section 4.2.3: What's the reason for including the Seq-GAN-BERT model? The section mentions that it might be useful in low-resource settings, however, it appears to be worse than most other systems and best when applied to English._\n\n**Author Response**\n\nThe Seq-GAN-BERT model has shown to perform well in extremely low resource settings (where we only have around 300-400 labeled sentences). However when the number of sentences increases, we observed that the performance saturates and does not reach very high numbers, as compared to other transformer architectures. Moreover, Seq-GAN-BERT was only tried before for English and 2 other Indian languages (Hindi and Marathi) and this paper is the first work for testing it on European languages. Since Indian languages have significant differences from the chosen European languages, that could be a reason why Seq-GAN-BERT underperforms. However, general numbers for all models are in the range of >85 F1 score, which indicates that our dataset can cater to a wide variety of sequence tagging algorithms.\n\n_Question 6: Section 5.2: It's unclear what this experiment tells us. It appears straightforward that if we evaluate an MT system that is trained on clean source text, the results should be worse if evaluated on disfluent source input. Wouldn't there be more informative downstream tasks to evaluate on? E.g. intent classification._\n\n**Author Response**\n\nLack of conversational translation datasets has led to most MT systems trained on fluent text. Our experiments show that such models if used in conversational settings do not perform well. By adding a DC model in the pipeline, which is often a smaller model with an incremental increase in latency, one can improve the downstream translations outputted by an MT system that does not adjust to conversational phenomena. Moreover, our dataset in German - English and Hindi - English can also be used to finetune conversational MT models.\n\n**Final Comments**\n\nThe authors sincerely thank the reviewer for their time to assess and evaluate our submission.\n\nWe apologize for the lack of clarification of these above points and would be happy to include these explanations in the revised version of this paper", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper contributes a valuable multilingual resource for four languages. It strengthens the data foundation for English and Hindi by increasing the amount of available data, as well as acts as an enabler for German and French by now allowing disfluency correction systems to be developed and analyzed.", "reasons_to_reject": "The paper contributes with a valuable multilingual dataset which is described in great detail. However, it is unclear how this dataset compares to existing datasets for English and Hindi - both in size and breadth (label types and coverage) as well as how it perhaps improves over challenges faced by prior datasets. Similarly, although the new languages German and French have no prior datasets to compare with, it is unclear how useful this additional data is. In particular, the results in Table 6 suggest that baselines can easily correct most disfluencies but that fluent utterances in English and French are hard to predict. A lack of analysis and comparison leaves it unclear if the dataset is trivial and how it makes a valuable addition to prior work.", "questions_to_authors": "- L166: How do you select sentences? Random, by hand, or using some heuristic?\n- Table 3: There appears to be a very high proportion of disfluent words in the dataset. Combined with Figure 2 it appears that the dataset contains very few negative samples (fluent/non disfluent utterances). Doesn't that mean that the dataset heavily biases systems to predict words as disfluent? Could this possibly explain the outliers in Table 6?\n- L196: You first write that there is only one annotator per language and then say that the *authors* take a majority vote to choose which samples to remove. Do all authors speak the four languages? How does that work?\n- Figure 2: The figure displays skewed distributions across disfluency types, with certain types being barely present. How does this impact the usefulness of the dataset? Wouldn't it make sense to have a more balanced dataset?\n- Section 4.2.3: What's the reason for including the Seq-GAN-BERT model? The section mentions that it might be useful in low-resource settings, however, it appears to be worse than most other systems and best when applied to English.\n- Section 5.2: It's unclear what this experiment tells us. It appears straightforward that if we evaluate an MT system that is trained on clean source text, the results should be worse if evaluated on disfluent source input. Wouldn't there be more informative downstream tasks to evaluate on? E.g. intent classification.\n- ", "paper_topic_and_main_contributions": "- This paper introduces and describes the creation of a disfluency correction dataset, named *DISCO*. The dataset comprises ~12k samples distributed across four languages: English, Hindi, German, and French, with about ~3k samples per language. \n- The dataset contributes with additional disfluency data for English and Hindi but is the first for German and French. \n- Utterances were sampled from an existing dataset of utterances made to AI agents (e.g. making notes, adding contacts) and annotated by a single annotator (one per language).\n- Utterances are annotated in a sequence-labeling manner, with each word being associated with a binary label (fluent/disfluent) and labeled with a disfluency type (filler/repetition/correction/false start/fluent)\n- Disfluency types are highly skewed with \"repetition\" being the majority class across all languages.\n- An array of baseline systems are evaluated on the dataset and F1 scores are reported in the high 90s for classifying disfluent utterances correctly, however, results are significantly worse for fluent utterances in English and French... This is attributed to unbalanced utterance types but it is unclear how this finding should be interpreted. \n- To show the downstream impact of applying disfluency correction, an MT experiment is conducted, reporting BLEU scores of an existing MT (not trained on disfluent source text) system when/when not applying disfluency correction on the Hi-En and De-En test sets. Results show that disfluency correction improves BLEU scores\n- The respective test sets are translated by language experts but no further information is documented about this translation data.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: We thank the authors for their submission. 

This paper introduces a new Disfluency Correction (DC) dataset in English, French, German, and Hindi. It then presents many methods (RNN, Transformers, etc.)

Reasons to Accept: 
- An practically valuable dataset for a potentially very useful under-studied but useful task (as pointed out by R1)
- Large range of baseline scores reported on the dataset (as pointed out by R1)
- The dataset is further showcasing the importance of DC for Machine Translation

Reasons to Reject:
- A single annotator per language: The paper would benefit from annotator agreement reported at least on a subsample of the data. 
- Better acknowledgment and discussion of related work. 

Suggestions: 
Evaluating the performance of rule-based predictions on DISCO would help us understand how challenging the dataset and task is (e.g., removing repetition with simple pattern matching.)

Typos Grammar Style And Presentation Improvements:
- Most tables do not follow the EMNLP template
Decision: Accept-Findings