Adversarial Training for Low-Resource Disfluency Correction
Year: 2023

Abstract:
Disfluencies commonly occur in conversational speech. Speech with disfluencies can result in noisy Automatic Speech Recognition (ASR) transcripts, which affects downstream tasks like machine translation. In this paper, we propose an adversarially-trained sequencetagging model for Disfluency Correction (DC) that utilizes a small amount of labeled real disfluent data in conjunction with a large amount of unlabeled data. We show the benefit of our proposed technique, which crucially depends on synthetically generated disfluent data, by evaluating it for DC in three Indian languagesBengali, Hindi, and Marathi (all from the IndoAryan family). Our technique also performs well in removing stuttering disfluencies in ASR transcripts introduced by speech impairments. We achieve an average 6.15 points improvement in F1-score over competitive baselines across all three languages mentioned. To the best of our knowledge, we are the first to utilize adversarial training for DC and use it to correct stuttering disfluencies in English, establishing a new benchmark for this task.

heading: 1 Introduction
text: Disfluencies are words that are part of spoken utterances but do not add meaning to the sentence. Disfluency Correction (DC) is an essential preprocessing step to clean disfluent sentences before passing the text through downstream tasks like machine translation (Rao et al., 2007; Wang et al., 2010). Disfluencies can be introduced in utterances due to two main reasons: the conversational nature of speech and/or speech impairments such as stuttering. In real-life conversations, humans frequently deviate from their speech plan, which can introduce disfluencies in a sentence (Dell et al., 1997). Stuttering speech consists of involuntary repetitions or prolongations of syllables which disturbs the fluency of speech.
Conversational disfluencies occur once every 17 words (Bortfeld et al., 2001) whereas a 2017 US
study1 shows that roughly 1% of the population stutters and predominantly consists of children. One out of every four children continues to suffer from this disorder lifelong. When such speech passes through an ASR system, readability of the generated transcript deteriorates due to the presence of disfluencies in speech (Jones et al., 2003).
Shriberg (1994) defines the surface structure of disfluent utterances as a combination of reparandum, interregnum and repair. The reparandum consists of the words incorrectly uttered by the speaker that needs correction or complete removal. The interregnum acknowledges that the previous utterance may not be correct, while repair contains the words spoken to correct earlier errors.
Data in DC is limited because of the time and resources needed to annotate data for training (Appendix A). Through this work2, we provide a method to create high-quality DC systems in low resource settings. Our main contributions are:
1. Improving the state-of-the-art in DC in Indian languages like Bengali, Hindi and Marathi by 9.19, 5.85 and 3.40 points in F1 scores, respectively, using a deep learning framework with adversarial training on real, synthetic and unlabeled data. 2. Creating an open-source stuttering English DC corpus comprising 250 parallel sentences 3. Demonstrating that our adversarial DC model can be used for textual stuttering correction
1https://www.nidcd.nih.gov/health/stuttering 2https://github.com/vineet2104/
AdversarialTrainingForDisfluencyCorrection
8112
with high accuracy (87.68 F1 score)

heading: 2 Related work
text: Approaches in DC can be categorized into noisy channel-based, parsing-based, and sequence tagging-based approaches. Noisy channel-based approaches rely on the following principle: a disfluent sentence Y can be obtained from a fluent sentence X by adding some noise. These models try to predict the fluent sentence X given the disfluent sentence Y (Honal and Schultz, 2004; Jamshid Lou and Johnson, 2017; Johnson and Charniak, 2004). Parsing-based approaches jointly predict the syntactic structure of the disfluent sentence along with its disfluent elements (Honnibal and Johnson, 2014; Jamshid Lou and Johnson, 2020; Rasooli and Tetreault, 2013; Wu et al., 2015; Yoshikawa et al., 2016). Sequence tagging-based approaches work on the following hypothesis: every word in a disfluent sentence can be marked as fluent/disfluent. These methods work best for shorter utterances and perform optimally for real-life conversational DC (Hough and Schlangen, 2015; Ostendorf and Hahn, 2013; Zayats et al., 2016). Moreover, sequence-tagging based methods require far less labeled data to perform well, compared to the other two methods. Our approach to DC focuses on treating it as a sequence tagging problem rather than a machine translation task. The objective is to accurately classify each word as either disfluent or fluent, and create fluent sentences by retaining only the fluent words. The lack of labeled data for DC in low-resource languages has prompted the use of semi-supervised methods and self-supervised techniques (Wang et al., 2018; Wang et al., 2021). DC has also been studied as a component in speech translation systems, and thus its effect has been analyzed in improving the accuracies of machine translation models (Rao et al., 2007; Wang et al., 2010). Synthetic data generation for DC has also received attention recently. These methods infuse disfluent elements in fluent sentences to create parallel data for training (Passali et al., 2022; Saini et al., 2020). Our work is an extension of Kundu et al. (2022), which creates the first dataset for DC in Bengali, Hindi and Marathi. We use this dataset to train our adversarial model to improve over the state-of-the-art in these languages. To the best of our knowledge, we are the first to model DC to correct stuttering ASR transcripts.

heading: 3 Types of Disfluencies
text: There are six broad types of disfluencies encountered in real life - Filled Pause, Interjection, Discourse Marker, Repetition or Correction, False Start and Edit. Although these are common in conversational speech, stuttering speech consists mainly of Filled Pauses and Repetitions. This section describes each type of disfluency and gives some examples in English.
1. Filled Pauses consist of utterances that have no semantic meaning. Example - What about the uh event? 2. Interjections are similar to filled pauses, but their inclusion in sentences indicates affirmation or negation. Example - Ugh, what a day it has been! 3. Discourse Markers help the speaker begin a conversation or keep turn while speaking. These words do not add semantic meaning to the sentence. Example - Well, we are going to the event. 4. Repetition or Correction covers the repetition of certain words in the sentence and correcting words that were incorrectly uttered. Example - If I can’t don’t go to the event today, it is not going to look good. 5. False Start occurs when previous chain of thought is abandoned, and new idea is begun. Example - Mondays dont work for me, how about Tuesday? 6. Edit refers to the set of words that are uttered to correct previous statements. Example - We need three tickets, I’m sorry, four tickets for the flight to California.

heading: 4 Architecture
text: The lack of labeled data for DC is a significant hurdle to developing state-of-the-art DC systems for low-resource languages. Passali et al. (2022), Saini et al. (2020) and Kundu et al. (2022) introduced data augmentation by synthesizing disfluencies in fluent sentences to generate parallel data. In this work, we propose a deep learning architecture that uses adversarial training to improve a BERT-based model’s token classification accuracy of whether a token is disfluent or not. Our proposed architecture uses real, synthetic and unlabeled data to improve classification performance.
Our model, Seq-GAN-BERT, is inspired by Croce et al. (2020), who first used a similar model
for sentence classification. It consists of three modules: a BERT-based encoder (Devlin et al., 2019), discriminator and generator. The encoder converts the input sequence X = (X1, X2, ...Xn) into encoded vector representations (Hreal). Simultaneously, the generator creates fake representations (Hfake) from Gaussian random noise (Z), mimicking the real data that passes through the encoder. The discriminator aims to solve a twopronged objective: i) predicting every word in the sentence to be disfluent or fluent and ii) determining whether the input from the generator comes from real or fake data.

heading: 4.1 Adversarial Training
text: The discriminator loss comprises two loss terms. The first loss is supervised by the token classification task, while the second loss is defined by the real/fake data identification task. Such adversarial training also allows the model to use unlabeled data during training. For unlabeled samples, only the real/fake data identification task is executed. The generator continuously improves during training and produces fake representations that resemble actual data. The competing tasks of the generator (to create better representations to fool the discriminator) and the discriminator (to
perform token classification for labeled sentences and real/fake identification) compels the MuRIL encoder to generate better representations of input sentences. The resulting high-quality representations allow the discriminator to identify disfluent words with a high accuracy.

heading: 5 Task 1: Few Shot DC in Indian Languages
text: To test our proposed architecture, we train the model on the few-shot DC task for Indian languages. The current state-of-the-art performance in Bengali, Hindi and Marathi DC is obtained by training a large multilingual transformer model using synthetic data created by injecting disfluencies in fluent sentences using rules (Kundu et al., 2022). We train our Seq-GAN-BERT model using the authors’ multilingual real and synthetic data.

heading: 5.1 Dataset
text: Our dataset consists of parallel disfluent-fluent sentences in three Indian languages. We use 300, 150 and 250 real disfluent sentences in Bengali, Hindi and Marathi, respectively and generate 1000 synthetic disfluent sentences in Bengali and 500 synthetic disfluent sentences each in Hindi and Marathi each by infusing disfluent elements in fluent transcriptions using a rule-based approach (Kundu et al., 2022). The synthetic data was created such that the percentage of disfluent words across 3 languages remains constant.

heading: 5.2 Text Processing and Training Details
text: Text pre-processing is performed by removing punctuations, lower-casing and creating wordlevel tokens for parallel sentences. The SeqGAN-BERT model uses a combination of labeled and unlabeled data comprising real and synthetically generated disfluent sentences in different languages. We try different combinations of monolingual and multilingual data. Our experiments show that the best model for Bengali uses real and synthetic Bengali sentences as labeled data and disfluent Hindi sentences as unlabeled data. The best model for Hindi uses real and synthetic Hindi sentences as labeled data and disfluent Bengali sentences as unlabeled data. The best model for Marathi uses real and synthetic Marathi sentences as labeled data and disfluent Bengali sentences as unlabeled data. The BERT-based transformer that we use as an encoder is the MuRIL model pretrained on English and many Indian lan-
guages (Khanuja et al., 2021). MuRIL representations for Indian languages are of superior quality compared to other multilingual Transformer-based models like mBERT (Devlin et al., 2019).

heading: 5.3 Evaluation
text: To evaluate our model, we train baselines for DC in zero-shot and few-shot settings. ZeroShot is based on Kundu et al. (2022). FewShot is based on training MuRIL on all real and synthetic data available in the chosen language, along with labeled data in a related Indian language (for Bengali, either Hindi or Marathi can act as a related Indian language). FewShotAdv is the Seq-GAN-BERT model without any unlabeled data. Although models like BiLSTM-CRF have been as alternatives to transformers for sequence tagging, direct finetuning often performs better (Ghosh et al., 2022). Performance of DC systems is usually measured with F1 scores (Ferguson et al., 2015; Honnibal and Johnson, 2014; Jamshid Lou and Johnson, 2017). Table 3 shows the comparison of various baselines against our model.
Our model, Seq-GAN-BERT with unlabeled sentences, performs better than the other baselines and establishes a new state-of-the-art for DC in Bengali, Hindi and Marathi. Our model benefits from adversarial training using both unlabeled data and multilingual training. Comparison of our model’s output with respect to the ZeroShot baseline is discussed in Table 2 (for more examples, refer to Appendix B). The observed precision and recall scores of these models during testing show that without adversarial training, the model performs with high precision but low recall. However, with adversarial training, the model improves its recall without compromising much on precision.
The zero-shot model (without adversarial training) classifies less words as disfluent but at a high accuracy, whereas the few-shot model (with adversarial training) correctly classifies more words as disfluent.

heading: 6 Task 2: Stuttering DC in English
text: We have already shown how our proposed architecture learns better semantic representations for DC using small amounts of manually annotated labeled data. In this section, we present a similar experiment in Stuttering DC (SDC). We define SDC as the task of removing disfluent elements in spoken utterances that are caused by stuttering speech impairment. Since this is the first attempt to model stuttering correction as disflu-
ency removal, we make our version of the existing dataset for stuttering publicly available for research purposes and provide various baseline comparisons. We show that our model generalizes well for this task and is able to remove disfluent elements in stuttering speech.

heading: 6.1 Dataset
text: The UCLASS dataset is created by transcribing audio interviews of 14 anonymous teenagers who stutter and consists of two released versions (Howell et al., 2004). Both versions of this corpus are available for free download and research. We create 250 disfluent-fluent parallel sentences from the available transcripts of such utterances. The dataset is released here3.

heading: 6.2 Processing & Training
text: We follow the same steps as before (section 5.2). Stuttered syllables are represented in the text, separated by a space delimiter and treated as a disfluent term. This gold-standard dataset is split into 150 sentences for training and 100 sentences for testing. The training sentences are used as labeled data for the model and unlabeled data from Switchboard (Godfrey et al., 1992) or Kundu et al. (2022) is used to facilitate multilingual training. Our model performs best when we use synthetic Bengali disfluent sentences as unlabeled data.

heading: 6.3 Evaluation
text: We use five baselines to evaluate our model’s performance. SupervisedGold uses the gold standard data and trains the MuRIL model for token classification. SupervisedGoldSWBD and SupervisedGoldLARD uses a combination of the gold standard dataset along with 1000 disfluent sentences from the Switchboard corpus and LARD dataset (Passali et al., 2021). AdversarialSWBD and AdversarialLARD uses the Seq-GAN-BERT to train on a combination of labeled sentences from gold standard corpus and unlabeled sentences from the Switchboard corpus and LARD dataset. Table 4 displays our results averaged over multiple seeds.
Our model outperforms all baselines. Improvement over AdversarialLARD shows the benefit of multilingual training. We also used synthetic Hindi or Marathi data while training, but achieved lower scores than the AdversarialLARD baseline.
3https://github.com/vineet2104/ AdversarialTrainingForDisfluencyCorrection
Summary of results: In this paper, we evaluate our proposed architecture for low-resource DC using two tasks: 1) DC in Indian languages and 2) Stuttering DC in English. Our model outperforms competitive baselines across both these tasks establishing a new state-of-the-art for Indian languages DC. The adversarial training in our model improves the representations of a BERT-based encoder for disfluent/fluent classification. We show that multilingual training benefits such tasks as the generator is trained to create better representations of fake data to fool the discriminator.

heading: 7 Conclusion
text: Adversarial training using unlabeled data can benefit disfluency correction when we have limited amounts of labeled data. Our proposed model can also be used to correct stuttering in ASR transcripts with high accuracy.
Future work lies in integrating speech recognition models like Whisper4 or wav2vec 2.0 (Baevski et al., 2020) to create end-to-end speechdriven DC models. It will also be insightful to see how this model transfers to other low-resource languages with different linguistic properties.

heading: 8 Acknowledgements
text: We would like to thank the anonymous reviewers and area chairs for their suggestions to strengthen the paper. This work was done as part of the Bahubhashak Pilot Project on Speech to Speech Machine Translation under the umbrella of National Language Technology Mission of Ministry of Electronics and IT, Govt. of India. We would
4https://cdn.openai.com/papers/whisper.pdf
also like to thank Nikhil Saini for valuable discussions during the course of this project.

heading: 9 Limitations
text: There are two main limitations of our work. Firstly, since there are no known baselines for Indian language DC except Kundu et al. (2022), other architectures might perform better than our model. Our claim that Seq-GAN-BERT tries to maximize the information gained from unlabeled sentences is supported by superior performance over baselines defined in this work and other related models. Secondly, due to the lack of good quality labeled datasets, our test sets contained only 100 sentences. However, we believe that the consistency of our high-performing models across languages and multiple seeded experiments presents a positive sign for DC in low-resource settings.

heading: 10 Ethics Statement
text: The aim of our work was to design an adversarial training-enabled token classification system that is able to correctly remove disfluencies in text. The datasets used in this work are publicly available and we have cited the sources of all the datasets that we have used.

heading: A Challenges in creating data for DC
text: There are three steps involved in creating data for DC - i) Transcribing the speech utterance, ii) Identifying disfluent elements in the transcript and iii) Creating the fluent sentence after removing disfluent utterances. Identifying disfluencies is not a straightforward task. Our observations show that, on average, it takes 2 minutes to create a pair of disfluent-fluent sentences for an average 15- second speech utterance.
Collecting data for SDC comes with its challenges. Currently available datasets only focus on speaker details and record stuttered speech for analysis. Since SDC requires speech to be transcribed and annotated, creating parallel sentences for training is difficult. We derive our dataset from open-source resources. However, to create manual data at a large scale, an appropriate recording environment must be designed where speakers who stutter can interact with others over various topics with skilled annotators listening and transcribing the audio. Thus, creating data for DC is a challenging task (Section 1) and we hope that our contributed dataset can facilitate further research in stuttering correction.

heading: B Case study: Analysing differences in the zero shot and few shot settings
text: In Indian languages DC, the ZeroShot baseline corresponds to a zero-shot method for DC, whereas our model is an adversarially trained few-shot method for DC. We perform qualitative comparisons across both these models to understand the difference through case studies from the test set. Table 5 shows our results. Our few-shot model qualitatively performs better than the zero-shot baseline in most cases and thus strengthens the results mentioned in Section 5.3.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Limitations have been mentioned as section 8 of the paper submitted

heading: A2. Did you discuss any potential risks of your work?
text: Not applicable. Since our paper is about disfluency correction through text, we do not anticipate any risks of our work or its potential use in other tasks.

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Yes, abstract and introduction summarize the paper’s main claims
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Section 5.1 describes the data we create

heading: 3 B1. Did you cite the creators of artifacts you used?
text: The authors of the dataset we use have been cited in Section 4.1 and Section 5.1

heading: 3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts?
text: It has been mentioned that the data we use is open source in sections 4.1 and 5.1

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? It has been mentioned that the data we use is open source and consistent with its intended use in sections 4.1 and 5.1

heading: 3 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any
text: information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? It has been mentioned in section 5.1 that the data we use and create is anonymous

heading: B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? Not applicable. The data we create is derived from an existing dataset that is open source and provides relevant documentation. We have cited the original dataset in section 4.1 and 5.1.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Relevant statistics have been mentioned in sections 4.2 and 5.2
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
C 3 Did you run computational experiments? Section 4.2 and Section 5.2

heading: C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Not applicable. Our model architecture does not compulsorily require any GPU support and thus is usable on many established frameworks

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Relevant details have been included in section 4.1, 4.2, 5.1 and 5.2

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Relevant details have been included in sections 4.3 and 5.3 and Appendix C

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Relevant details have been included in sections 4 and 5
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.

heading: D1. Did you report the full text of instructions given to participants, including e.g., screenshots,
text: disclaimers of any risks to participants or annotators, etc.? Not applicable. Left blank.

heading: D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students)
text: and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Not applicable. Left blank.

heading: D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to
text: crowdworkers explain how the data would be used? Not applicable. Left blank.

heading: D4. Was the data collection protocol approved (or determined exempt) by an ethics review board?
text: Not applicable. Left blank.

heading: D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data?
text: Not applicable. Left blank.

heading: Future Work
text: Future work lies in integrating speech recognition models like Whisper4 or wav2vec 2. 0 (Baevski et al. , 2020) to create end-to-end speechdriven DC models. It will also be insightful to see how this model transfers to other low-resource languages with different linguistic properties