SlowGuess/NAACL_2022_OCR huggingface dataset:
In this section we present a summary of both the non-incremental and incremental performance of our streaming model against that the baselines. We also present an analysis of the types of errors
and average amount of waiting time the streaming model incurs.
5.1 Incremental Performance
Table 1 shows both the incremental and non-incremental evaluation metrics. Our proposed streaming BERT $_{\text{SV}}$  model achieved a  $9\%$  increase $^{1}$  in streaming  $F_{1}$  over both of the baselines (with lookahead = 0), as well as a  $71\%$  and  $72\%$  decrease in edit overhead compared to the non-streaming models trained on full sequences and all prefixes, respectively. Despite being trained with a different architecture and loss objective, the streaming model does not sacrifice its non-incremental performance, yielding a final output  $F_{1}$  score that is only one point less than its non-streaming counterparts. Generally speaking, when the streaming model does output a prediction, it classifies tokens as disfluent less often than the non-streaming models with zero LA window, achieving much higher precision (P) and marginally lower recall (R), resulting in a model that "flickers" less frequently. However, this does contribute to a slightly higher time-to-detection score compared to the baselines with zero lookahead, since the streaming model is generally less aggressive but more precise with outputting disfluent predictions. When compared to the models with fixed lookahead (the lower half of Table 1), however, the streaming model always achieves lower TTD while achieving significantly lower waiting time and comparable accuracy and
<sup>1</sup>All percentages mentioned in this section are computed as a percentage of the original number, rather than as a difference in percentage points.
Type of disfluency	Average wait time
Repair	0.74
Fluent	0.15
Interregnum	1.06
Reparandum	0.76
Edit	0.14
Repair onset	0.46
Table 2: The streaming model's average waiting time (in number of tokens) for each type of token (as categorized in Marcus et al. (1999)) encountered in each prefix fed to the model for the Switchboard validation set. For a more fine-grained analysis, we separate the repair onset (the first word in the repair phrase) from the rest of the words in the repair. The category "Edit" consists of all edit terms that are not interregna (i.e. not inside of a repair structure).
Effect of lookahead window size We also evaluated the performance of the non-streaming baseline models with fixed lookahead window sizes of 1 and 2 tokens, as shown in the lower half of Table 1. In line with what has been reported in past work (Madureira and Schlangen, 2020; Buß and Schlangen, 2011), the size of the lookahead window scales directly with the accuracy and stability and inversely with the latency of the model. However, the streaming model has comparable streaming  $F_{1}$  and edit overhead scores as the non-streaming models with  $LA = 1$ , even though the streaming model has  $79\%$  less average wait time. This indicates that the streaming model is able to correctly classify tokens sooner and with more stability than the baseline models that have  $LA = 1$ . Although the models with  $LA = 2$  improve marginally on accuracy and stability over the models with  $LA = 1$ , the streaming model continues to have lower TTD and AWT but comparable  $F_{1}$  and EO when compared to the models with  $LA = 2$ .
The utility of dynamic lookahead The results in Table 1 also reveal some insights into which parts of the model design and training scheme are more important for streaming performance and efficiency. Merely training a non-streaming model on prefixes of the training examples appears to have minimal effect on  $F_{1}$ , precision, and recall, but does somewhat improve the TTD score. We hypothesize that this is largely the result of training on a data distribution that more closely resembles
the test distribution. Adding the extra wait classifier head and latency cost term in the training objective yields the greatest improvements in both precision and stability, as seen in the differences in  $F_{1}$ , P, and EO values between the BERT $_{\mathrm{SV}}$  model trained on all prefixes and the streaming BERT $_{\mathrm{SV}}$  model.
When to wait Since the streaming model can abstain from outputting predictions for arbitrarily long suffixes of the input, it incurs waiting time - an average of 0.21 tokens more than the non-streaming models with 0 lookahead. Table 2 shows that the streaming model abstains the most when encountering interregna and reparanda, waiting for approximately 1.06 and 0.76 more tokens, respectively. Given that it is easier to identify a disfluency once the entire reparandum and interregnum have been observed, it follows that the model's predictions may be more uncertain for reparanda and interregna upon first consumption, thus incurring the highest average waiting times. An example of the model's incremental outputs for a disfluency structure is shown in Table 4. For correctly classified disfluent tokens, the streaming model also has a higher TTD, likely because the non-incremental models are more aggressive in predicting disfluent labels (while making more errors) than the streaming model. Still, this TTD is lower than all the models with fixed lookahead windows.
Figure 3: Token-wise misclassification rates for each type of disfluency annotation (as categorized in Marcus et al. (1999)) across all three evaluated models on the Switchboard validation set. If the model abstained from making a prediction on a particular token, we did not count this as an error.
Figure 3 shows an error analysis on the models' predictions. We computed the percentage of the
Model	Training Scheme	Incremental metrics
EO↓	FTD↓
BERTLARGE (Rohanian and Hough, 2021)	All prefixes	0.60	0.31
BERTSV	Full sequences	0.30	0.79
BERTSV	All prefixes	0.32	0.84
Streaming BERTSV	All prefixes	0.09	0.11
Table 3: A comparison of the EO and FTD metrics of our baselines (BERT $_{SV}$  trained on full sequences and all prefixes), our streaming BERT $_{SV}$  model, and Rohanian and Hough (2021)'s incrementalized BERT $_{LARGE}$  model on the Switchboard test set. The arrows near each metric represent the desirable direction of the result – for both of the metrics, lower numbers are more desirable.
time that the streaming model misclassified a token, counting each incidence of the token across each time step separately. All models achieved the lowest misclassification rates on fluent and edit tokens, and the highest misclassification rates on reparanda tokens. For tokens that were fluent, edit terms, or part of a repair or repair onset, the streaming model achieved significantly lower misclassification rates than the baselines. However, all three models performed comparably on interregna and reparanda. Since we measure misclassification rate on every token at every time step, the high misclassification rates on reparanda are expected as it is often not feasible to detect a reparandum until an interregnum or repair onset has been seen.
Accuracy versus latency In comparison to the baseline models with 0 lookahead, the streaming model makes the largest tradeoffs in accuracy versus latency for repair onsets and repairs, as shown by Figure 3 and Table 2. While the streaming model incurs average additional wait times of 0.74 and 0.46 tokens for repairs and repair onsets respectively, its misclassification rates are also approximately  $85\%$  and  $82\%$  less than the baseline models on repairs and repair onsets respectively. In addition, Table 1 demonstrates that the streaming model still achieves comparable  $F_{1}$  and greater stability (lower EO) in comparison to the non-streaming baselines with lookahead 1, despite having an average wait time that is  $79\%$  shorter.
5.3 Comparison with Competitor Baselines
As shown in Table 3, in comparison with the BERTLarge-based prophecy decoding model proposed in Rohanian and Hough (2021), our streaming model achieves state-of-the-art stability (85% decrease in EO) and latency (65% decrease in
$\mathrm{FTD})^2$  , despite having far fewer parameters.
We have introduced a streaming BERT-based Transformer model that is capable of balancing accuracy with latency by simultaneously making token-level disfluency predictions and dynamically deciding how large of a lookahead window to use. Our approach improves both streaming accuracy and output stability on an incremental disfluency detection task. Furthermore, it incurs very low average latency in comparison with non-incremental BERT models of the same size. Lastly, our model requires minimal lookahead beyond disfluent regions and achieves state-of-the-art edit overhead and first-time-to-detection scores compared to past work (Rohanian and Hough, 2021).
While the main focus of this paper has been on developing a fast, accurate, and stable streaming model for disfluency detection, our approach is general enough to be used in other incremental tagging models of linguistic phenomena that benefit from the right context for optimal accuracy. In future work we are interested in applying this approach to tasks such as real-time punctuation prediction and incremental parsing.
Furthermore, the streaming model's efficiency is limited by its non-autoregressive nature and training via restart incrementality. Future work should also explore how to apply a dynamic lookahead window without re-computing the predictions on all the previous inputs and to rely on fewer prefixes during training.
${}^{2}$  In addition to shorter word-level latency metrics presented in the results, the runtime latency of the  ${\mathrm{{BERT}}}_{\mathrm{{SV}}}$  model is  ${80}\%$  lower than that of  ${\mathrm{{BERT}}}_{\mathrm{{BASE}}}$  (Rocholl et al.,2021).
While our work does not introduce a new dataset, it does depend on a training dataset that was collected from fluent English-speaking, able-bodied human subjects. If deployed in a real-world application, this model would likely perform noticeably worse for users who speak with non-American accents or speech impediments. Transcripts for these users could be disproportionately noisy and the streaming model's average wait time would likely also be longer. Care should be taken to assess the sensitivity and robustness of such a model to non-fluent or non-American English prior to deployment. This model should also be used very cautiously in situations where mistakenly eliding fluent portions of speech from the captions or transcript could incur dire consequences, such as in an emergency call center.
We thank Johann Rocholl, Colin Cherry, Dan Liebling, Noah Murad, and members of the Google Research Speech Intelligence team for valuable feedback and discussion. We also thank the Google Student Researcher and research internship programs for supporting this work. Lastly, we also thank the anonymous reviewers for their thorough and helpful comments.
Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dandelion Mane, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaojiang Zheng. 2015. TensorFlow: Large-scale machine learning on heterogeneous systems. Software available from tensorflow.org.
Nguyen Bach and Fei Huang. 2019. Noisy BiLSTM-based models for disfluency detection. In Proceedings of Interspeech 2019, pages 4230-4234.
Timo Baumann Okko Buß and David Schlangen. 2011. Evaluation and optimisation of incremental processors. Dialogue & Discourse, 2(1):113-141.
Eugene Charniak and Mark Johnson. 2001. Edit detection and parsing for transcribed speech. In Second Meeting of the North American Chapter of the Association for Computational Linguistics.
Qian Chen, Mengzhe Chen, Bo Li, and Wen Wang. 2020. Controllable time-delay transformer for real-time punctuation prediction and disfluency detection. In ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 8069-8073.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
J.J. Godfrey, E.C. Holliman, and J. McDaniel. 1992. Switchboard: telephone speech corpus for research and development. In [Proceedings] ICASSP-92: 1992 IEEE International Conference on Acoustics, Speech, and Signal Processing, volume 1, pages 517-520 vol.1.
Daniel Golovin, Benjamin Solnik, Subhodeep Moitra, Greg Kochanski, John Karro, and D. Sculley. 2017. Google vizier: A service for black-box optimization. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '17, page 1487-1495, New York, NY, USA. Association for Computing Machinery.
Aditya Gupta, Jiacheng Xu, Shyam Upadhyay, Diyi Yang, and Manaal Faruqui. 2021. Disfl-QA: A benchmark dataset for understanding disfluencies in question answering. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 3309-3319, Online. Association for Computational Linguistics.
Hany Hassan, Lee Schwartz, Dilek Hakkani-Tur, and Gokhan Tur. 2014. Segmentation and disfluency removal for conversational speech translation. In Fifteenth Annual Conference of the International Speech Communication Association.
Julian Hough and David Schlangen. 2015. Recurrent Neural Networks for Incremental Disfluency Detection. In *Interspeech* 2015, pages 849-853.
Julian Hough and David Schlangen. 2017. Joint, incremental disfluency detection and utterance segmentation from speech. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 326-336, Valencia, Spain. Association for Computational Linguistics.
Paria Jamshid Lou and Mark Johnson. 2020. End-to-end speech recognition and disfluency removal. In *Findings of the Association for Computational Linguistics: EMNLP* 2020, pages 2051-2061, Online. Association for Computational Linguistics.
Patrick Kahardipraja, Brielen Madureira, and David Schlangen. 2021. Towards incremental transformers: An empirical analysis of transformer models for incremental nlu.
Brielen Madureira and David Schlangen. 2020. Incremental processing in the age of non-incremental encoders: An empirical assessment of bidirectional models for incremental NLU. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 357-374, Online. Association for Computational Linguistics.
Mitchell P. Marcus, Beatrice Santorini, Mary Ann Marcinkiewicz, and Ann Taylor. 1999. Treebank-3.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1-67.
Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don't know: Unanswerable questions for SQuAD. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 784-789, Melbourne, Australia. Association for Computational Linguistics.
Johann C. Rocholl, Vicky Zayats, Daniel D. Walker, Noah B. Murad, Aaron Schneider, and Daniel J. Liebling. 2021. Disfluency Detection with Unlabeled Data and Small BERT Models. In Proc. Interspeech 2021, pages 766-770.
Morteza Rohanian and Julian Hough. 2020. Reframing incremental deep language models for dialogue processing with multi-task learning. In Proceedings of the 28th International Conference on Computational Linguistics, pages 497-507, Barcelona, Spain (Online). International Committee on Computational Linguistics.
Morteza Rohanian and Julian Hough. 2021. Best of both worlds: Making high accuracy non-incremental transformer-based disfluency detection incremental. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 3693-3703, Online. Association for Computational Linguistics.
Igor Shalyminov, Arash Eshghi, and Oliver Lemon. 2017. Challenging neural dialogue models with natural data: Memory networks fail on incremental phenomena. In Proceedings of the 21st Workshop on the Semantics and Pragmatics of Dialogue - Full Papers, Saarbrücken, Germany. SEMDIAL.
Igor Shalyminov, Arash Eshghi, and Oliver Lemon. 2018. Multi-task learning for domain-general spoken disfluency detection in dialogue systems. In Proceedings of the 22nd Workshop on the Semantics and Pragmatics of Dialogue - Full Papers, Aix-en-Provence, France. SEMDIAL.
Elizabeth Shriberg, Rebecca Bates, and Andreas Stolcke. 1997. A prosody only decision-tree model for disfluency detection. In *Fifth European Conference on Speech Communication and Technology*.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumont, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online. Association for Computational Linguistics.
Vicky Zayats, Mari Ostendorf, and Hannaneh Hajishirzi. 2016. Disfluency detection using a bidirectional LSTM. CoRR, abs/1604.03209.
Sanqiang Zhao, Raghav Gupta, Yang Song, and Denny Zhou. 2021. Extremely small BERT models from mixed-vocabulary training. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 2753-2759, Online. Association for Computational Linguistics.
Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In 2015 IEEE International Conference on Computer Vision (ICCV), pages 19-27.
Simon Zwarts, Mark Johnson, and Robert Dale. 2010. Detecting speech repairs incrementally using a noisy channel approach. In Proceedings of the 23rd International Conference on Computational Linguistics (Coling 2010), pages 1371-1378, Beijing, China. Coling 2010 Organizing Committee.
We provide more detailed formulas here for each of our evaluation metrics. For a given input utterance  $x$ , let  $x[j]$  be the  $j$ -th token of  $x$ ,  $y[j]$  be the gold label (either 0 for fluent or 1 for disfluent) for  $x[j]$ ,  $x[: i]$  be the  $i$ -th prefix of  $x$  (i.e., the first  $i$  tokens of  $x$ ),  $f(x[: i])[j]$  be the predicted label for  $x[j]$  after the model has consumed prefix  $i$ ,  $D$  be the entire dataset that we are evaluating predictive performance for, and  $|D|$  be the size of  $D$ .
Accuracy metrics Our definitions of the streaming true positives, true negatives, false positives, and false negatives are:
$$
T P _ {\text {s t r e a m i n g}} = | \{(i, j) | y [ j ] = 1, f (x [: i ]) [ j ] = 1 \} |
$$
$$
T N _ {\text {s t r e a m i n g}} = | \{(i, j) | y [ j ] = 0, f (x [: i ]) [ j ] = 0 \} |
$$
$$
F P _ {\text {s t r e a m i n g}} = | \{(i, j) | y [ j ] = 0, f (x [: i ]) [ j ] = 1 \} |
$$
$$
F N _ {\text {s t r e a m i n g}} = | \{(i, j) | y [ j ] = 1, f (x [: i ]) [ j ] = 0 \} |
$$
Similar to the traditional definitions, streaming precision, recall, and  $F_{1}$  are computed as:
$$
P _ {\text {s t r e a m i n g}} = \frac {T P _ {\text {s t r e a m i n g}}}{T P _ {\text {s t r e a m i n g}} + F P _ {\text {s t r e a m i n g}}}
$$
$$
R _ {\text {s t r e a m i n g}} = \frac {T P _ {\text {s t r e a m i n g}}}{T P _ {\text {s t r e a m i n g}} + F N _ {\text {s t r e a m i n g}}}
$$
$$
F 1 _ {\text {s t r e a m i n g}} = 2 \times \frac {P _ {\text {s t r e a m i n g}} \times R _ {\text {s t r e a m i n g}}}{P _ {\text {s t r e a m i n g}} + R _ {\text {s t r e a m i n g}}}
$$
Stability metrics
- Edit overhead (EO) (Buß and Schlangen, 2011): To calculate edit overhead, we need to first identify, for each prefix  $x[:i]$  of a given input  $x$ , which tokens  $x[j]$  have predictions  $f(x[:i])[j]$  that differ from the model's prediction in the previous prefix  $f(x[:i-1])[j]$ . Denoting the cardinality of this set as  $E(x)$  (for the number of edits the model makes on  $x$ ), we have:
$$
E (x) = | \{(i, j) | f (x [: i ]) [ j ] \neq f (x [: i - 1 ]) [ j ] \} |
$$
Then we can compute EO as follows:
$$
E O = \frac {1}{| D |} \sum_ {x \in D} \frac {E (x)}{| x |}
$$
where  $|x|$  is the number of tokens in  $x$ .
Latency metrics
- Time-to-detection (TTD) (Hough and Schlangen, 2017): Since time-to-detection (TTD) is measured only on disfluent tokens that are eventually predicted as such by the model, we need to first define the set of tokens in a given example  $x$  that are true positives at some point:
$$
\begin{array}{l} T P (x) = \{x [ k ] \mid y [ k ] = 1, \\ \exists i: f (x [ i ]) [ k ] = 1 \} \\ \end{array}
$$
Then for a given token  $x[k] \in x$ , the detection time (DT) can be calculated as:
$$
D T (x [ k ]) = \min  _ {i} \left\{i \mid f (x [: i ]) [ k ] = 1 \right\} - k
$$
It follows that the TTD for the entire dataset  $D$  is the average  $DT$  for all disfluent tokens that are eventually detected for all  $x \in D$ :
$$
T T D = \frac {1}{m} \sum_ {x [ k ] \in T P (x), x \in D} D T (x [ k ])
$$
$$
m = | D | \sum_ {x \in D} | T P (x) |,
$$
the total number of disfluent tokens in the dataset that are eventually detected by the model.
- First time to detection (FTD) (Zwarts et al., 2010; Rohanian and Hough, 2021): Similar to TTD, this metric is only measured on disfluent tokens that are eventually detected by the model. Then given some  $x[k] \in TP(x)$ , let  $RI(x[k])$  represent the index of the first token in the repair that follows  $x[k]$ . Since we are measuring detection time from the start of a gold standard repair instead, the detection time becomes:
$$
\begin{array}{l} D T (x [ k ]) = \min  _ {i} \{i | f (x [ i ]) [ k ] = 1 \} \\ - R I (x [ k ]) \\ \end{array}
$$
The rest of the formula for the FTD is similar to that of the TTD:
$$
F T D = \frac {1}{m} \sum_ {x [ k ] \in T P (x), x \in D} D T (x [ k ])
$$
- Average waiting time (AWT): Suppose that given an input token  $x[k]$ , the model can abstain from making a prediction (which occurs both with the streaming model and with the fixed lookahead models). We denote this outcome as  $y[k] = \emptyset$ . To compute AWT, we first calculate the first prediction time (FPT) for a given token  $x[k]$ ,
$$
F P T (x [ k ]) = \arg \min _ {i} \{i | f (x [: i ]) [ k ] \neq \emptyset \},
$$
i.e. the first time step  $i$  in which the model outputs a prediction for token  $x[k]$ . Then the AWT is
$$
A W T = \frac {1}{m} \sum_ {x [ k ] \in x, x \in D} F P T (x [ k ]) - k,
$$
$$
m = | D | \sum_ {x \in D} | x |,
$$
the total number of tokens in the dataset.
A.2 Model Training and Hyperparameter Tuning