SlowGuess/ACL_2020_OCR huggingface dataset:
We compare the performance of the self-attentive parser with state-of-the-art disfluency detection models. As shown in Table 8, our model has the best f-score. We also compare our model with prior work that reported EDITED, INTJ and PRN word f-score for disfluency detection and find that our model has the best performance (see Table 9). Compared to Wang et al. (2018) which uses GANs to leverage additional unlabelled data and Bach and Huang (2019) which leverages synthetic data, our model significantly improves the recall. This demonstrates that standard techniques such as self-training and ensembling are as good or better than these specialized, complex approaches.
Table 7: Parse precision  $P$  ,recall  $R$  and f-score  $F$  for all constituent spans on the Switchboard test set. *Text+prosody model.  $\mathrm{P} = \mathrm{P}\left( \mathrm{S}\right) ,\mathrm{R} = \mathrm{R}\left( \mathrm{S}\right)$  and  $\mathrm{F} = \mathrm{F}\left( \mathrm{S}\right)$  .
Disfluency (E)	P	R	F
Tran et al. (2018)	-	-	76.7
Tran et al. (2018)*	-	-	77.5
Jamshid Lou et al. (2018)5	89.5	80.0	84.5
Zayats et al. (2016)	91.8	80.6	85.9
Jamshid Lou and Johnson (2017)	-	-	86.8
Wang et al. (2016)	91.6	82.3	86.7
Jamshid Lou et al. (2019)	81.7	92.8	86.9
Wang et al. (2017)	91.1	84.1	87.5
Dong et al. (2019)	94.5	84.1	89.0
This work (single model)	86.7	91.9	89.2
This work (ensemble of 4)	87.5	93.8	90.6
Table 8: EDITED word precision  $P$ , recall  $R$  and f-score  $F$  on the Switchboard test set. *Text+prosody model.  $\mathrm{P} = \mathrm{P}\left( {\mathrm{W}}_{\mathrm{E}}\right) ,\mathrm{R} = \mathrm{R}\left( {\mathrm{W}}_{\mathrm{E}}\right)$  and  $\mathrm{F} = \mathrm{F}\left( {\mathrm{W}}_{\mathrm{E}}\right)$  .
Disfluency (EIP)	P	R	F
Wang et al. (2018)	92.1	90.2	91.1
Bach and Huang (2019)	94.7	89.8	92.2
This work (single model)	92.2	96.6	94.3
This work (ensemble of 4)	92.5	97.2	94.8
Table 9: EDITED, INTJ and PRN (EIP) word precision  $P$ , recall  $R$  and f-score  $F$  on the Switchboard test set.  $\mathrm{P = P(W_{EIP})}$ ,  $\mathrm{R = R(W_{EIP})}$  and  $\mathrm{F = F(W_{EIP})}$ .
5.1 Qualitative Results
We conduct a qualitative analysis on the Switchboard dev set to characterize the disfluencies that
$^{5}$ https://github.com/pariajm/deep-disfluency-detection
the baseline model cannot detect but the self-trained one can. We provide representative examples in Table 5. In general, the self-trained model is better at detecting long complex corrections (# 1-4), restarts (# 5) and stutter-like repetitions (# 6). It also does a better job of discriminating fluent repetitions and fluent parallel structures from repetition and correction types of disfluency (# 7 and 8). Figure 3 depicts a sentence parsed by the baseline and the self-trained self-attentive parser, where the self-trained model correctly predicts all disfluency EDITED nodes. As explained in Section 3, we do not use an external POS tagger, so POS tags are not available when parsing from raw text. That's why all preterminal labels in Figure 3 are shown by a dummy token i.e. UNK.
We introduced a new state-of-the-art for joint disfluency detection and constituency parsing of transcribed speech. We showed that self-training and ensembling are effective methods for improving disfluency detection. A qualitative analysis of the results also indicated that self-training is helpful for detecting complicated types of disfluencies, including corrections and restarts. In future work, we intend to explore the idea of self-training for parsing written texts. We also aim at integrating syntactic parsing and self-training more closely with automatic speech recognition. The first step is to develop parsing models that parse ASR output, rather than speech transcripts.
We would like to thank the anonymous reviewers for their insightful comments and suggestions. This research was supported by a Google award through the Natural Language Understanding Focused Program, by a CSIRO's DATA61 Top-up Scholarship, and under the Australian Research Councils Discovery Projects funding scheme (project number DP160102156).
Nguyen Bach and Fei Huang. 2019. Noisy BiLSTM-based models for disfluency detection. In Proceedings of Interspeech 2019, pages 4230-4234, Graz, Austria.
Eugene Charniak and Mark Johnson. 2001. Edit detection and parsing for transcribed speech. In Proceedings of the  $2^{nd}$  Meeting of the North American
Chapter of the Association for Computational Linguistics on Language Technologies, pages 118-126, Stroudsburg, USA.
Do Kook Choe and Eugene Charniak. 2016. Parsing as language modeling. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2331-2336, Austin, USA.
Christopher Cieri, David Graff, Owen Kimball, Dave Miller, and Kevin Walker. 2004. *Fisher English training speech part 1 transcripts LDC2004T19*. Published by: Linguistic Data Consortium, Philadelphia, USA.
Christopher Cieri, David Graff, Owen Kimball, Dave Miller, and Kevin Walker. 2005. *Fisher English training speech part 2 transcripts LDC2005T19*. Published by: Linguistic Data Consortium, Philadelphia, USA.
Kevin Clark, Minh-Thang Luong, Christopher D. Manning, and Quoc Le. 2018. Semi-supervised sequence modeling with cross-view training. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1914-1925, Brussels, Belgium.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4171-4186, Minneapolis, USA.
Qianqian Dong, Feng Wang, Zhen Yang, Wei Chen, Shuang Xu, and Bo Xu. 2019. Adapting translation models for transcript disfluency detection. In Proceedings of the 2019 AAAI Conference on Artificial Intelligence, pages 6351-6358, Honolulu, USA.
Kira Droganova, Filip Ginter, Jenna Kanerva, and Daniel Zeman. 2018. Mind the gap: Data enrichment in dependency parsing of elliptical constructions. In Proceedings of the Second Workshop on Universal Dependencies, pages 47-54, Brussels, Belgium.
Chris Dyer, Adhiguna Kuncoro, Miguel Ballesteros, and Noah A. Smith. 2016. Recurrent neural network grammars. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 199-209, San Diego, California. Association for Computational Linguistics.
Daniel Fried, Nikita Kitaev, and Dan Klein. 2019. Cross-domain generalization of neural constituency parsers. In Proceedings of the  $57^{th}$  Annual Meeting of the Association for Computational Linguistics, pages 323-330, Florence, Italy.
Daniel Fried, Mitchell Stern, and Dan Klein. 2017. Improving neural parsing by disentangling model combination and reranking effects. In Proceedings of
the  $55^{\text{th}}$  Annual Meeting of the Association for Computational Linguistics, pages 161-166, Vancouver, Canada.
John Godfrey and Edward Holliman. 1993. Switchboard-1 Release 2 LDC97S62. Published by: Linguistic Data Consortium, Philadelphia, USA.
Matthew Honnibal and Mark Johnson. 2014. Joint incremental disfluency detection and dependency parsing. Transactions of the Association for Computational Linguistics, 2(1):131-142.
Paria Jamshid Lou, Peter Anderson, and Mark Johnson. 2018. Disfluency detection using auto-correlational neural networks. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4610-4619, Brussels, Belgium.
Paria Jamshid Lou and Mark Johnson. 2017. Disfluency detection using a noisy channel model and a deep neural language model. In Proceedings of the  $55^{\text{th}}$  Annual Meeting of the Association for Computational Linguistics, pages 547-553, Vancouver, Canada.
Paria Jamshid Lou, Yufei Wang, and Mark Johnson. 2019. Neural constituency parsing of speech transcripts. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2756-2765, Minneapolis, USA.
Mark Johnson and Eugene Charniak. 2004. A TAG-based noisy channel model of speech repairs. In Proceedings of the  $42^{nd}$  Annual Meeting on Association for Computational Linguistics, pages 33-39, Barcelona, Spain.
Mark Johnson, Eugene Charniak, and Matthew Lease. 2004. An improved model for recognizing disfluencies in conversational speech. In Proceedings of Rich Transcription Workshop.
Jeremy Kahn, Matthew Lease, Eugene Charniak, Mark Johnson, and Mari Ostendorf. 2005. Effective use of prosody in parsing conversational speech. In Proceedings of the Conference on Human Language Technology and Empirical Methods in Natural Language Processing, pages 233-240, Tallinn, Estonia.
Nikita Kitaev, Steven Cao, and Dan Klein. 2019. Multilingual constituency parsing with self-attention and pre-training. In Proceedings of the  $57^{th}$  Annual Meeting of the Association for Computational Linguistics, pages 3499–3505, Florence, Italy.
Nikita Kitaev and Dan Klein. 2018. Constituency parsing with a self-attentive encoder. In Proceedings of the  $56^{\text{th}}$  Annual Meeting of the Association for Computational Linguistics, pages 2676–2686, Melbourne, Australia.
Matthew Lease and Mark Johnson. 2006. Early deletion of fillers in processing conversational speech. In Proceedings of the Human Language Technology
Conference of the North American Chapter of the ACL, pages 73-76, Stroudsburg, USA.
Mitchell Marcus, Beatrice Santorini, Mary Ann Marcinkiewicz, and Ann Taylor. 1999. Treebank-3 LDC99T42. Published by: Linguistic Data Consortium, Philadelphia, USA.
David McClosky, Eugene Charniak, and Mark Johnson. 2006. Effective self-training for parsing. In Proceedings of the 2006 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 152-159, New York City, USA.
Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2227-2237, New Orleans, Louisiana.
Mohammad Sadegh Rasooli and Joel Tetreault. 2013. Joint parsing and disfluency detection in linear time. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 124-129, Seattle, USA.
Elizabeth Shriberg. 1994. Preliminaries to a theory of speech disfluencies. Ph.D. thesis, Department of Psychology, University of California, Berkeley, USA.
Trang Tran, Shubham Toshniwal, Mohit Bansal, Kevin Gimpel, Karen Livescu, and Mari Ostendorf. 2018. Parsing speech: A neural approach to integrating lexical and acoustic-prosodic information. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 69-81, New Orleans, USA.
Trang Tran, Jiahong Yuan, Yang Liu, and Mari Ostendorf. 2019. On the role of style in parsing speech with neural models. In Proceedings the  $19^{\text{th}}$  Annual Conference of the International Speech Communication Association, pages 4190-4194, Graz, Austria.
Feng Wang, Wei Chen, Zhen Yang, Qianqian Dong, Shuang Xu, and Bo Xu. 2018. Semi-supervised disfluency detection. In Proceedings of the  $27^{th}$  International Conference on Computational Linguistics, pages 3529-3538, Santa Fe, USA.
Shaolei Wang, Wanxiang Che, and Ting Liu. 2016. A neural attention model for disfluency detection. In Proceedings of the  $26^{th}$  International Conference on Computational Linguistics, pages 278-287, Osaka, Japan.
Shaolei Wang, Wanxiang Che, Yue Zhang, Meishan Zhang, and Ting Liu. 2017. Transition-based disfluency detection using LSTMs. In Proceedings of
the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2785-2794, Copenhagen, Denmark.
Masashi Yoshikawa, Hiroyuki Shindo, and Yuji Matsumoto. 2016. Joint transition-based dependency parsing and disfluency detection for automatic speech recognition texts. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 1036-1041, Austin, USA.
Victoria Zayats, Mari Ostendorf, and Hannaneh Hajishirzi. 2016. Disfluency detection using a bidirectional LSTM. In Proceedings of the  $16^{\text{th}}$  Annual Conference of the International Speech Communication Association, pages 2523-2527, San Francisco, USA.