Venue: ACL 2020

# Improving Disfluency Detection by Self-Training a Self-Attentive Model

Paria Jamshid Lou\({}^{1}\) and Mark Johnson\({}^{2,1}\)

\({}^{1}\)Department of Computing, Macquarie University

\({}^{2}\)Oracle Digital Assistant, Oracle Corporation

\({}^{1}\)paria.jamshid-lou@hdr.mq.edu.au

\({}^{2}\)mark.mj.johnson@oracle.com

###### Abstract

Self-attentive neural syntactic parsers using contextualized word embeddings (e.g. ELMo or BERT) currently produce state-of-the-art results in joint parsing and disfluency detection in speech transcripts. Since the contextualized word embeddings are pre-trained on a large amount of unlabeled data, using additional unlabeled data to train a neural model might seem redundant. However, we show that self-training -- a semi-supervised technique for incorporating unlabeled data -- sets a new state-of-the-art for the self-attentive parser on disfluency detection, demonstrating that self-training provides benefits orthogonal to the pre-trained contextualized word representations. We also show that ensembling self-trained parsers provides further gains for disfluency detection.

## 1 Introduction

Speech introduces challenges that do not appear in written text, such as the presence of disfluencies. Disfluency refers to any interruptions in the normal flow of speech, including false starts, corrections, repetitions and filled pauses. Shriberg (1994) defines three distinct parts of a speech disfluency, referred to as the _reparamdum_, the _interregnum_ and the _repair_. As illustrated in the example below, the reparamdum _The first kind of invasion of_ is the part of the utterance that is replaced or repaired, the interregnum _uh I mean_ (which consists of a filled pause _uh_ and a discourse marker _I mean_) is an optional part of the disfluency, and the repair _the first type of privacy_ replaces the reparamdum. The fluent version is obtained by removing the reparamdum and the interregnum.

Jamshid Lou et al. (2019) showed that a self-attentive constituency parser achieves state-of-the-art results for joint parsing and disfluency detection. They observed that because the Switchboard

Figure 1: A parse tree from the Switchboard corpus, where reparamdum _The first kind of invasion of_, filled pause _uh_ and discourse marker _I mean_ are dominated by EDITED, INTJ and PRN nodes.



trees include both syntactic constituency nodes and EDITED nodes that indicate disfluency, training a parser to predict the Switchboard trees can be regarded as multi-task learning (where the tasks are syntactic parsing and identifying disfluencies). In this paper, we extend the multi-task learning in Jamshid Lou et al. (2019) to explore the impact of self-training McClosky et al. (2006) and ensembling Kitaev et al. (2019) on the performance of the self-attentive parser. We aim to answer two questions about the state-of-the-art self-attentive parser:

* _Does self-training improve the performance of the self-attentive parser on disfluency detection?_ Self-training is a semi-supervised technique for incorporating unlabeled data into a new model, where an existing model trained on manually labeled (i.e. gold) data is used to label unlabeled data. The automatically (i.e. silver) labeled data are treated as truth and combined with the gold labeled data to re-train a new model McClosky et al. (2006); Choe and Charniak (2016). Since neural models use rich representations of language pre-trained on a large amount of unlabeled data Peters et al. (2018); Devlin et al. (2019), we might expect that self-training adds no new information to the self-attentive parser. Surprisingly, however, we find that **self-training improves disfluency detection f-score of the BERT-based self-attentive parser**, demonstrating that self-training provides benefits orthogonal to the pre-trained contextualized embeddings.
* _Does ensembling improve disfluency detection in speech transcripts?_ Ensembling is a commonly used technique for improving parsing where scores of multiple instances of the same model trained on the same or different data are combined at inference time Dyer et al. (2016); Fried et al. (2017); Kitaev et al. (2019). We expect ensembling parsers to improve the performance of the model on disfluency detection, too. We show **ensembling four self-trained parsers (using different BERT word representations) via averaging their span label scores increases disfluency detection f-score** in comparison with a single self-trained parser.

## 2 Related Work

Parsing speech transcripts is challenging for conventional syntactic parsers, mainly due to the presence of disfluencies. In disfluent sentences, the relation between reparandum and repair is different from other words in the sentence. The repair is usually a "rough copy" of the reparamundum, using the same or similar words in roughly the same word order1Charniak and Johnson (2001). Designed to capture tree-like structures, conventional syntactic parsers fail to detect "rough copies" which are strong indicators of disfluency. Moreover, the reparandum and repair often do not form a syntactic phrase, which makes detecting the reparandum even harder. For these reasons, specialized disfluency detection models were developed to remove disfluencies prior to parsing Charniak and Johnson (2001); Kahn et al. (2005); Lease and Johnson (2006) or special mechanisms were added to parsers to handle disfluencies Rasooli and Tetreault (2013); Honnibal and Johnson (2014); Yoshikawa et al. (2016). Conventional parsing based models can use the syntactic location of the disfluency as a feature in a reranker Johnson et al. (2004). A similar gain can be achieved in neural models by training a joint parsing and disfluency detection model. In this multi-task learning setting, syntactic information helps the neural model detect disfluencies more accurately Jamshid Lou et al. (2019).

Footnote 1: For example in Figure 1, the reparandum _The first kind of invasion of_ and the repair _the first type of privacy_ are “rough copies” of each other.

State-of-the-art results for disfluency detection have been reported for Transformer models using contextualized embeddings (e.g. ELMo and BERT) Jamshid Lou et al. (2019); Tran et al. (2019); Dong et al. (2019). The self-attention mechanism of the Transformer is apparently effective for capturing "rough copy" dependencies between words. A recent study shows that prosody slightly improves the parsing performance of the self-attentive model over the text-only model, especially in long sentences Tran et al. (2019). In this paper, we use a self-attentive model for joint disfluency detection and constituency parsing.

Disfluency detection models are usually trained and evaluated on the Switchboard corpus. Switchboard is the largest disfluency annotated dataset. However, only 5.9% of the words in the Switchboard are disfluent Charniak and Johnson (2001).

 

To mitigate the scarcity of labeled data, some studies have leveraged additional data by using: (i) contextualized embeddings pre-trained on enormous amount of unlabeled data [18, 23] and (ii) synthetic data generated by adding noise in the form of disfluencies to fluent sentences (e.g. repeating, deleting or inserting words in a sentence) [45, 60, 79]. By contrast, this paper focuses on self-training, which is a simple semi-supervised technique that has been effective in different NLP tasks, including parsing [14, 15, 16]. To our best knowledge, this is the first work that investigates self-training a neural disfluency detection model.

Another technique commonly used for improving parsing is ensembling. Ensembling is a model combination method, where scores of multiple models (they can be the same or different models, trained on the same or different data, with different random initializations) are combined in some way [15, 16, 17]. The state-of-the-art for parsing written text is an ensemble of four BERT-based self-attentive parsers, where the parsers are combined by averaging their span label scores [14]. While ensembling is widely used in parsing, it has not been investigated for disfluency detection. In this paper, we also explore the impact of ensembling several parsing based disfluency detection models on disfluency detection performance.

## 3 Model

Following Jamshid Lou et al. (2019), we use a self-attentive constituency parser for joint disfluency detection and syntactic parsing2. The parsing model is based on the architecture introduced by Kitaev and Klein (2018), which is state-of-the-art for (i) parsing written texts [14, 15], (ii) parsing transcribed speech [23], and (iii) joint parsing and disfluency detection [16].

Footnote 2: The code is available at: [https://github.com/pariajm/joint-disfluency-detection-and-parsing](https://github.com/pariajm/joint-disfluency-detection-and-parsing)

The self-attentive parser assigns a score \(s(T)\) to each tree \(T\) by calculating the sum of the potentials on its labeled constituent spans:

\[s(T)=\sum_{(i,j,l)\in T}s(i,j,l) \tag{1}\]

where \(s(i,j,l)\) is the score of a constituent beginning at string position \(i\) ending at position \(j\) with label \(l\). The input to the parser is a sequence of vectors corresponding to the sequence of words in a sentence followed by one or more self-attention layers. For each span \((i,j)\), a hidden vector \(h_{ij}\) is constructed by subtracting the representations of the start and end of the span. A span classifier, including two fully connected layers followed by a non-linearity, assigns labeling scores \(s(i,j,..)\) to each span. Then, the highest scoring parse tree is found for a given sentence as follows:

\[\hat{T}=\operatorname*{argmax}_{T}s(T) \tag{2}\]

using a modified CYK algorithm. The parser introduced in Kitaev and Klein (2018) relies on an external POS tagger to predict pretermal labels, but because the parser's accuracy does not decrease when no external POS tagger is used, we use their parser here without an external POS tagger (hence, all the pretermal labels are UNK). For more details, see Kitaev and Klein (2018).

### Contextualized Embeddings

We incorporate BERT [15] in our self-attentive parser by fine-tuning the parameters as part of the training process. Following Kitaev et al. (2019), we apply a learned projection matrix on the output of BERT to project the vectors to our desired dimensionality. The representations are then fed into the parser. BERT learns the representations for sub-word units, so to extract the word representations, we consider the representations of the last sub-word unit for each word in the sentence [14].

### Self-Training

We train the self-attentive parser on the Penn Treebank-3 Switchboard corpus which contains gold disfluency labeled parse trees [16, 14]. Using the trained model, we parse unlabeled data and add the silver parse trees to the gold Switchboard training data and re-train the self-attentive parser using the enlarged training set. The unlabeled data we use include Fisher Speech Transcripts Part 1 [Cieri 

et al., 2004) and Part 2 (Cieri et al., 2005). Table 1 summarizes the different datasets used to train the self-attentive parser.

Footnote 3: Words tagged as “XX” or words ending in “..”

## 4 Experiments

Following Charniak and Johnson (2001), we split the Switchboard into training, dev and test sets as follows: training data consists of the sw[23]\(*\).mrg files, dev data consists of the sw4[5-9]\(*\).mrg files and test data consists of the sw4[0-1]\(*\).mrg files. All partial words4 and punctuations are removed from the data, as they are not available in realistic ASR applications (Johnson and Charniak, 2004).

Footnote 4: Words tagged as “XX” or words ending in “..”

### Baseline

Our baseline is the self-attentive parser trained on the gold Switchboard corpus with BERT word representations. The BERT-based parser is the current state-of-the-art, providing a very strong baseline for our work. We trained different versions of the baseline parser using four different BERT models, namely BERTBASE[cased] and BERTLARGE[cased][uncased], and then selected the best model i.e. BERTBASE[cased] on the Switchboard dev set. We also tuned the hyperparameters by optimizing for performance on parsing EDITED nodes \(F(S_{E})\). Preliminary experiments on the Switchboard dev set showed that the hyperparameters given by Kitaev et al. (2019) perform well; therefore, this is what we used here. Since random seeds lead to different results, in this paper we report average scores across 5 runs of each model initialized with different random seeds.

### Evaluation Metrics

We evaluate the self-attentive parser in terms of parsing accuracy, as well as disfluency detection. Since certain words are identified as EDITED in the parse tree, we can measure how well a parser classifies words as EDITED. We can also evaluate how accurately the parser can identify all disfluency words, i.e., the words dominated by EDITED, INTJ or PRN nodes. Therefore, we report _precision_ (P), _recall_ (R) and _f-score_ (F) for both constituent spans (S) and word positions (W), where each word position is treated as labeled by all the constituents containing that word. We also report the result for subsets of constituent spans and word positions: (i) S\({}_{\text{E}}\), the set of constituent spans labeled EDITED, (ii) W\({}_{\text{E}}\), the set of word positions dominated by one or more EDITED nodes, and (iii) W\({}_{\text{EP}}\), the set of word positions dominated by one or more EDITED, INTJ or PRN nodes. For more details, see Jamshid Lou et al. (2019).

### Varying Amount of Silver Training Data

To find the optimal proportion of additional silver training data, we select \(n\) percent (ranging from \(10\%\) to \(90\%\)) of the training data in each mini-batch from silver parse trees and the rest from the gold ones. This has the same effect as reweighting the main gold corpus as in McClosky et al. (2006). The results for using different proportions of the silver parse trees are presented in Figure 2. The BERT-based parser self-trained with 40% silver Fisher trees and 60% gold Switchboard trees is our best model. In other words, for a batch size of 30, in each mini-batch 12 parse trees come from the silver Fisher data and 18 parse trees from the gold Switchboard. All self-training results in this paper use this proportion of gold and silver parse trees.

Figure 2: EDITED node f-score \(F(S_{E})\) of the BERT-based self-attentive parser as a function of percentages of training data in each mini-batch sourced from silver Fisher trees.

\begin{table}
\begin{tabular}{l l c c} \hline \hline
**Dataset** & **Labels** & **\# Sens** & **\# Words** \\ \hline SWB & gold & 98k & 733k \\ Fisher & silver & 835k & 14m \\ \hline \end{tabular}
\end{table}
Table 1: Summary of the datasets used to train the self-attentive parser.

 



### Does self-training improve the performance of the self-attentive parser?

Tables 2 and 3 compare the baseline and the self-trained parser in terms of parsing and disfluency detection. The parser self-trained on the silver Fisher data increases parsing and disfluency detection performance, indicating the BERT-based model benefits from additional silver labeled data. Self-training is especially effective for recognizing EDITED disfluency nodes (\(1.5\%\) increase in f-score). Only \(5.9\%\) of the words in the Switchboard are disfluent, and BERT is only trained on fluent texts such as books and Wikipedia, so the baseline parser may be starved of disfluent training examples. As a result, self-training on a corpus of conversational speech may compensate for the scarcity of disfluent gold data. To explore this, we tried self-training on a wide variety of fluent clean datasets, including Gigaword 5 (which is an unlabelled newswire corpus) and WSJ and Brown (which include gold parse trees of written text), but the performance did not improve significantly. This suggests that the parser benefits more from additional in-domain (i.e. conversational) silver data than additional out-of-domain (i.e. written) silver/gold data. Moreover, if we learn the embeddings as part of training instead of using pre-trained BERT, EDITED word f-score would drop from 90.9% to 86.4% and self-training on Fisher leads to little improvement (0.2% increase in EDITED word f-score compared to 1.5% improvement when using BERT). This suggests that self-training works well when the baseline model is powerful enough to predict accurate silver labels.

To further investigate the influence of self-training on disfluency detection, we randomly select \(100\) sentences containing disfluencies from the Switchboard dev set. We categorize disfluencies into _repetition_, _correction_ and _restart_ according to Shriberg's (1994) typology of speech repairs. _Repetitions_ are repairs where the reparamdum and repair portions of the disfluency are identical, while _corrections_ are where the reparandum and repairs differ (which are much harder to detect). _Restarts_ are where the speaker abandons a sentence and starts a new one (i.e. the repair is empty). As Table 4 shows, the self-trained parser outperforms the baseline in detecting all types of disfluency. It especially has a better performance at detecting corrections and restarts which are more challenging types of disfluency in comparison with repetitions.

### Does ensembling parsers improve disfluency detection?

We investigate the impact of ensembling on the performance of the self-attentive parser, where we combine parsers by averaging their span label scores as follows:

\[s_{ensemble}(i,j,l)=\frac{1}{4}\sum_{n=1}^{4}s_{n}(i,j,l) \tag{3}\]

We tried different ensembling of parsers and the best result was achieved when we trained the baseline parser four times using four BERT word rep

\begin{table}
\begin{tabular}{p{71.1pt} p{34.1pt} p{34.1pt} p{34.1pt}} \hline
**Model** & **Rep.** & **Cor.** & **Res.** & **All** \\ \hline Baseline & \(97.0\) & \(80.6\) & \(82.0\) & \(89.2\) \\ Self-trained & \(97.3\) & \(88.6\) & \(87.8\) & \(92.9\) \\ \hline \end{tabular}
\end{table}
Table 4: EDITED word f-score \(F(W_{\text{E}})\) for different types of disfluency on a subset of the Switchboard dev set containing \(158\) disfluent structures — including \(90\) repetitions (Rep.), \(54\) corrections (Cor.) and \(14\) restarts (Res.).

\begin{table}
\begin{tabular}{p{71.1pt} p{34.1pt} p{34.1pt}} \hline
**Parsing** & **F(S\({}_{\text{E}}\))** & **F(S\({}_{\text{EIP}}\))** & **F(S)** \\ \hline Baseline & \(89.2\) & \(95.6\) & \(93.5\) \\ Self-trained & \(90.7\) & \(96.2\) & \(93.9\) \\ \hline \end{tabular}
\end{table}
Table 2: Parse f-score for EDITED node _F(S\({}_{\text{E}}\))_, for EDITED, INTJ and PRN nodes _F(SELP)_ and for all constituent spans _F(S)_ on the Switchboard dev set for the baseline parser and the parser trained on the silver Fisher data.

\begin{table}
\begin{tabular}{p{71.1pt} p{34.1pt} p{34.1pt}} \hline
**Disfluency** & **F(W\({}_{\text{E}}\))** & **F(W\({}_{\text{EIP}}\))** \\ \hline Baseline & \(90.9\) & \(95.3\) \\ Self-trained & \(92.4\) & \(96.0\) \\ \hline \end{tabular}
\end{table}
Table 3: EDITED word f-score _F(W\({}_{\text{E}}\))_, EDITED, INTJ and PRN word f-score _F(W\({}_{EIP}\))_ on the Switchboard dev set for the baseline parser and the parser trained on the silver Fisher data.



resentations, namely \({\text{BERT}_{\text{BASE [cased]uncased}}}\) and \({\text{BERT}_{\text{LARGE [cased]uncased}}}\), and combined the results at inference time Kitaev et al. (2019). The ensembled models not only reflect variations of different pre-trained representations but also the randomness in initialization of the models. As shown in Table 6, ensembling and self-training both improve the performance of the baseline single model on parsing and detecting EDITED disfluency nodes. Self-training is more effective than ensembling, especially for EDITED node detection. The best results are reported for ensembling the best of the self-trained parsers for each of different BERT models from the 5 random restarts4.

Footnote 4: We also tried ensembling all 20 versions of the self-trained parser initialized with different random seeds. The results were 0.1% worse than the ensemble of the four best self-trained parsers.

## 7 Conclusion

We have proposed a novel method to generate the 



## 5 Results

We compare the performance of our best model with previous work on the Switchboard test set. As demonstrated in Table 7, our model outperforms prior work in parsing. The parsing result for our model is higher than Tran et al. (2018) which utilizes prosodic cues, as well as text based features.

We compare the performance of the self-attentive parser with state-of-the-art disfluency detection models. As shown in Table 8, our model has the best f-score. We also compare our model with prior work that reported EDITED, INTJ and PRN word f-score for disfluency detection and find that our model has the best performance (see Table 9). Compared to Wang et al. (2018) which uses GANs to leverage additional unlabelled data and Bach and Huang (2019) which leverages synthetic data, our model significantly improves the recall. This demonstrates that standard techniques such as self-training and ensembling are as good or better than these specialized, complex approaches.

### Qualitative Results

We conduct a qualitative analysis on the Switchboard dev set to characterize the disfluencies that

\begin{table}
\begin{tabular}{l c c c} \hline \hline
**Parsing (S)** & **P** & **R** & **F** \\ \hline Tran et al. (2018) & \(-\) & \(-\) & \(87.9\) \\ Tran et al. (2018)* & \(-\) & \(-\) & \(88.5\) \\ Jamshid Lou et al. (2019) & \(92.4\) & \(92.9\) & \(92.7\) \\ Tran et al. (2019) & \(-\) & \(-\) & \(92.8\) \\ Tran et al. (2019)* & \(-\) & \(-\) & \(93.0\) \\ This work (single model) & \(93.2\) & \(93.8\) & \(93.5\) \\ This work (ensemble of 4) & \(93.6\) & \(94.2\) & \(93.9\) \\ \hline \hline \end{tabular}
\end{table}
Table 7: Parse precision \(P\), recall \(R\) and f-score \(F\) for all constituent spans on the Switchboard test set. *Text+prosody model. P=P(S), R=R(S) and F=F(S).

Figure 3: A sentence from the Switchboard dev set parsed by the baseline model (left) and by the self-trained model (right). The parse tree obtained by the self-trained model is the same as the gold parse tree.



the baseline model cannot detect but the self-trained one can. We provide representative examples in Table 5. In general, the self-trained model is better at detecting long complex corrections (# 1-4), restarts (# 5) and stutter-like repetitions (# 6). It also does a better job of discriminating fluent repetitions and fluent parallel structures from repetition and correction types of disfluency (# 7 and 8). Figure 3 depicts a sentence parsed by the baseline and the self-trained self-attentive parser, where the self-trained model correctly predicts all disfluency EDITED nodes. As explained in Section 3, we do not use an external POS tagger, so POS tags are not available when parsing from raw text. That's why all preterminal labels in Figure 3 are shown by a dummy token i.e. UNK.

## 6 Conclusion

We introduced a new state-of-the-art for joint disfluency detection and constituency parsing of transcribed speech. We showed that self-training and ensembling are effective methods for improving disfluency detection. A qualitative analysis of the results also indicated that self-training is helpful for detecting complicated types of disfluencies, including corrections and restarts. In future work, we intend to explore the idea of self-training for parsing written texts. We also aim at integrating syntactic parsing and self-training more closely with automatic speech recognition. The first step is to develop parsing models that parse ASR output, rather than speech transcripts.

## Acknowledgments

We would like to thank the anonymous reviewers for their insightful comments and suggestions. This research was supported by a Google award through the Natural Language Understanding Focused Program, by a CSIRO's DATA61 Top-up Scholarship, and under the Australian Research Councils Discovery Projects funding scheme (project number DP160102156).

## References

* Bach and Huang (2019) Nguyen Bach and Fei Huang. 2019. Noisy BiLSTM-based models for disfluency detection. In _Proceedings of Interspeech 2019_, pages 4230-4234, Graz, Austria.
* Charniak and Johnson (2001) Eugene Charniak and Mark Johnson. 2001. Edit detection and parsing for transcribed speech. In _Proceedings of the \({}^{2m}\) Meeting of the North American Chapter of the Association for Computational Linguistics on Language Technologies_, pages 118-126, Stroudsburg, USA.
* Choe and Charniak (2016) Do Kook Choe and Eugene Charniak. 2016. Parsing as language modeling. In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_, pages 2331-2336, Austin, USA.
* Cieri et al. (2004) Christopher Cieri, David Graff, Owen Kimball, Dave Miller, and Kevin Walker. 2004. _Fisher English training speech part 1 transcripts LDC2004T19_. Published by: Linguistic Data Consortium, Philadelphia, USA.
* Cieri et al. (2005) Christopher Cieri, David Graff, Owen Kimball, Dave Miller, and Kevin Walker. 2005. _Fisher English training speech part 2 transcripts LDC2005T19_. Published by: Linguistic Data Consortium, Philadelphia, USA.
* Clark et al. (2018) Kevin Clark, Minh-Thang Luong, Christopher D. Manning, and Quoc Le. 2018. Semi-supervised sequence modeling with cross-view training. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_, pages 1914-1925, Brussels, Belgium.
* Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 4171-4186, Minneapolis, USA.
* Dong et al. (2019) Qianqian Dong, Feng Wang, Zhen Yang, Wei Chen, Shuang Xu, and Bo Xu. 2019. Adapting translation models for transcript disfluency detection. In _Proceedings of the 2019 AAAI Conference on Artificial Intelligence_, pages 6351-6358, Honolulu, USA.
* Droganova et al. (2018) Kira Droganova, Filip Ginter, Jenna Kanerva, and Daniel Zeman. 2018. Mind the gap: Data enrichment in dependency parsing of elliptical constructions. In _Proceedings of the Second Workshop on Universal Dependencies_, pages 47-54, Brussels, Belgium.
* Dyer et al. (2016) Chris Dyer, Adhiguna Koncoro, Miguel Ballesteros, and Noah A. Smith. 2016. Recurrent neural network grammars. In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 199-209, San Diego, California. Association for Computational Linguistics.
* Fried et al. (2019) Daniel Fried, Nikita Kitaev, and Dan Klein. 2019. Cross-domain generalization of neural constituency parsers. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 323-330, Florence, Italy.
* Fried et al. (2017) Daniel Fried, Mitchell Stern, and Dan Klein. 2017. Improving neural parsing by disentangling model combination and reranking effects. In _Proceedings of 

the 55th Annual Meeting of the Association for Computational Linguistics_, pages 161-166, Vancouver, Canada.
* Godfrey and Holliman (1993) John Godfrey and Edward Holliman. 1993. _Switchboard-1 Release 2 LDC97562_. Published by: Linguistic Data Consortium, Philadelphia, USA.
* Honnibal and Johnson (2014) Matthew Honnibal and Mark Johnson. 2014. Joint incremental disfluency detection and dependency parsing. _Transactions of the Association for Computational Linguistics_, 2(1):131-142.
* Lou et al. (2018) Paria Jamshid Lou, Peter Anderson, and Mark Johnson. 2018. Disfluency detection using auto-correlational neural networks. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_, pages 4610-4619, Brussels, Belgium.
* Lou and Johnson (2017) Paria Jamshid Lou and Mark Johnson. 2017. Disfluency detection using a noisy channel model and a deep neural language model. In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics_, pages 547-553, Vancouver, Canada.
* Lou et al. (2019) Paria Jamshid Lou, Yufei Wang, and Mark Johnson. 2019. Neural constituency parsing of speech transcripts. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 2756-2765, Minneapolis, USA.
* Johnson and Charniak (2004) Mark Johnson and Eugene Charniak. 2004. A TAG-based noisy channel model of speech repairs. In _Proceedings of the 42nd Annual Meeting on Association for Computational Linguistics_, pages 33-39, Barcelona, Spain.
* Johnson et al. (2004) Mark Johnson, Eugene Charniak, and Matthew Lease. 2004. An improved model for recognizing disfluencies in conversational speech. In _Proceedings of Rich Transcription Workshop_.
* Kahn et al. (2005) Jeremy Kahn, Matthew Lease, Eugene Charniak, Mark Johnson, and Mari Ostendorf. 2005. Effective use of prosody in parsing conversational speech. In _Proceedings of the Conference on Human Language Technology and Empirical Methods in Natural Language Processing_, pages 233-240, Tallinn, Estonia.
* Kitaev et al. (2019) Nikita Kitaev, Steven Cao, and Dan Klein. 2019. Multilingual constituency parsing with self-attention and pre-training. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 3499-3505, Florence, Italy.
* Kitaev and Klein (2018) Nikita Kitaev and Dan Klein. 2018. Constitutory parsing with a self-attentive encoder. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics_, pages 2676-2686, Melbourne, Australia.
* Lease and Johnson (2006) Matthew Lease and Mark Johnson. 2006. Early deletion of fillers in processing conversational speech. In _Proceedings of the Human Language Technology Conference of the North American Chapter of the ACL_, pages 73-76, Stroudsburg, USA.
* Marcus et al. (1999) Mitchell Marcus, Beatrice Santorini, Mary Ann Marcinkiewicz, and Ann Taylor. 1999. _Treebank-3 LDC99T42_. Published by: Linguistic Data Consortium, Philadelphia, USA.
* McClosky et al. (2006) David McClosky, Eugene Charniak, and Mark Johnson. 2006. Effective self-training for parsing. In _Proceedings of the 2006 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 152-159, New York City, USA.
* Peters et al. (2018) Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 2227-2237, New Orleans, Louisiana.
* Rasooli and Tetreault (2013) Mohammad Sadegh Rasooli and Joel Tetreault. 2013. Joint parsing and disfluency detection in linear time. In _Proceedings of the Conference on Empirical Methods in Natural Language Processing_, pages 124-129, Seattle, USA.
* Shriberg (1994) Elizabeth Shriberg. 1994. _Preliminaries to a theory of speech disfluencies_. Ph.D. thesis, Department of Psychology, University of California, Berkeley, USA.
* Tran et al. (2018) Trang Tran, Shubham Toshniwal, Mohit Bansal, Kevin Gimpel, Karen Livescu, and Mari Ostendorf. 2018. Parsing speech: A neural approach to integrating lexical and acoustic-prosodic information. In _Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 69-81, New Orleans, USA.
* Tran et al. (2019) Trang Tran, Jiahong Yuan, Yang Liu, and Mari Ostendorf. 2019. On the role of style in parsing speech with neural models. In _Proceedings the 19th Annual Conference of the International Speech Communication Association_, pages 4190-4194, Graz, Austria.
* Wang et al. (2018) Feng Wang, Wei Chen, Zhen Yang, Qianqian Dong, Shuang Xu, and Bo Xu. 2018. Semi-supervised disfluency detection. In _Proceedings of the 27th International Conference on Computational Linguistics_, pages 3529-3538, Santa Fe, USA.
* Wang et al. (2016) Shaolei Wang, Wanxiang Che, and Ting Liu. 2016. A neural attention model for disfluency detection. In _Proceedings of the 26th International Conference on Computational Linguistics_, pages 278-287, Osaka, Japan.
* Wang et al. (2017) Shaolei Wang, Wanxiang Che, Yue Zhang, Meishan Zhang, and Ting Liu. 2017. Transition-based disfluency detection using LSTMs. In _Proceedings of 

the 2017 Conference on Empirical Methods in Natural Language Processing_, pages 2785-2794, Copenhagen, Denmark.
* Yoshikawa et al. (2016) Masashi Yoshikawa, Hiroyuki Shindo, and Yuji Matsumoto. 2016. Joint transition-based dependency parsing and disfluency detection for automatic speech recognition texts. In _Proceedings of the Conference on Empirical Methods in Natural Language Processing_, pages 1036-1041, Austin, USA.
* Zayats et al. (2016) Victoria Zayats, Mari Ostendorf, and Hannaneh Hajishirzi. 2016. Disfluency detection using a bidirectional LSTM. In _Proceedings of the 16th Annual Conference of the International Speech Communication Association_, pages 2523-2527, San Francisco, USA.