SlowGuess/NAACL_2022_OCR huggingface dataset:
New York University ac5968@nyu.edu
Vicky Zayats
Daniel D. Walker
vzayats@google.com danwalkeriv@google.com
Dirk Padfield
padfield@google.com
In modern interactive speech-based systems, speech is consumed and transcribed incrementally prior to having disfluencies removed. This post-processing step is crucial for producing clean transcripts and high performance on downstream tasks (e.g. machine translation). However, most current state-of-the-art NLP models such as the Transformer operate non-incrementally, potentially causing unacceptable delays. We propose a streaming BERT-based sequence tagging model that, combined with a novel training objective, is capable of detecting disfluencies in real-time while balancing accuracy and latency. This is accomplished by training the model to decide whether to immediately output a prediction for the current input or to wait for further context. Essentially, the model learns to dynamically size its lookahead window. Our results demonstrate that our model produces comparably accurate predictions and does so sooner than our baselines, with lower flicker. Furthermore, the model attains state-of-the-art latency and stability scores when compared with recent work on incremental disfluency detection.
Many modern Natural Language Understanding (NLU) applications (e.g. transcribers, digital voice assistants, and chatbots) use streaming Automatic Speech Recognition (ASR) systems that incrementally consume speech, offering real-time transcription and predictions with minimal delay. However, these systems are often challenged by the presence of disfluencies, which are unintentional speech disruptions such as "um", "no I meant", and "I I I I think," that occur naturally in spontaneous speech. Disfluencies not only hurt the readability of ASR transcripts, but also erode model performance on downstream tasks, such as machine translation (Hassan et al., 2014) and question answering
(Gupta et al., 2021). Indeed, even state-of-the-art models such as BERT (Devlin et al., 2019) and T5 (Raffel et al., 2020) exhibit significant drops in performance (as much as 28 and 20 F1 points, respectively) on the SQuAD-v2 question-answering benchmark (Rajpurkar et al., 2018) when disfluencies are inserted into the questions (Gupta et al., 2021). Past work has shown that a prohibitively large amount of data is needed to train an end-to-end dialogue model that is robust to the presence of disfluencies (Shalyminov et al., 2017). As a result, modern ASR pipelines typically contain a separate post-processing step that detects and removes disfluencies from the transcript, which has been shown to perform better than end-to-end ASR models that generate fluent text from disfluent speech (Jamshid Lou and Johnson, 2020).
Shriberg et al. (1997) introduced the following disfluency schema components that are widely used in disfluency detection research: the reparandum (spoken segment intended to be removed), the interruption point (marked as “+”), the repair (spoken segment that comes as a replacement to the reparandum, of which the first word is known as the repair onset), and the interregnum (material that appears between the reparandum and repair). An example of this annotation schema is shown in Figure 1. Usually the disfluency detection task involves identifying and removing the reparandum portion of the disfluency. One of the most popular approaches that targets disfluency detection is the usage of sequence tagging models such as finetuned BERT (Bach and Huang, 2019; Rohanian and Hough, 2021) or LSTM (Zayats et al., 2016; Rohanian and Hough, 2020).
Another challenge in disfluency detection is the fact that most interactive speech- or text-based applications consume input incrementally, producing predictions one word at a time, rather than in entire sentences. However, recent state-of-the-art pretrained language models such as BERT have largely
*Work completed as part of the Student Researcher program at Google.
Human Language Technologies, pages 827 - 838
Figure 1: An example of disfluency annotation.
been designed for non-incremental processing and are trained only to output predictions on complete input utterances. Using a non-incremental model in an interactive setting produces undesirable delays, since downstream applications must wait for the user to finish their entire utterance before making any decisions.
To address the goal of streaming disfluency detection, recent work has focused on adapting non-incremental models for streaming settings. Madureira and Schlangen (2020) demonstrated that BERT-based models can adequately process incremental input for a variety of sequence tagging tasks when trained on partial sequences, although performance on full sequences suffers. Rohanian and Hough (2021) applied both the truncated training and prophecy generation strategies from (Madureira and Schlangen, 2020) to a  $\mathrm{BERT}_{\mathrm{LARGE}}$  model, achieving state-of-the-art performance on streaming metrics among incremental systems. Notably, both these approaches employ the delay strategy of a fixed lookahead window - a short amount of right context that the model can "peek" at when making its prediction on the current token (Buß and Schlangen, 2011). Although a larger lookahead window can boost accuracy and stability, it also incurs extra delay (by definition).
In the task of incremental disfluency detection, a lookahead window is likely most useful for reparanda, since it is often nearly impossible to identify a reparandum without knowing whether it is followed by an interregnum or repair. However, this extra right context may be much less informative for fluent tokens. Guided by this insight, we extend the past research by training a BERT-based model to dynamically decide how much lookahead context to use. For each new input token that the model consumes, the model can choose to either immediately output a label for that token or to wait for further input before making its prediction. We also design a novel training objective that accounts for both the cross-entropy and latency costs incurred by delaying inference.
In our experiments we explore the trade-offs be
tween accuracy, latency, and output stability for both partial and complete results. To our knowledge, this is the first work to adapt the BERT architecture and training objective to balance accuracy and latency in a streaming sequence tagging task.
The contributions of our paper are as follows: first, we propose a new model architecture and training objective for streaming sequence tagging tasks. This method involves fine-tuning a pretrained BERT model to decide when to immediately output predictions and when to wait for further input- temporarily abstaining from producing a prediction. Secondly, we show that this model achieves high accuracy in incremental settings with state-of-the-art latency and stability, all with a model architecture that is  $\sim 35$  times smaller than  $\mathrm{BERT}_{\mathrm{BASE}}$  (Zhao et al., 2021). We demonstrate that the model continues to perform competitively in non-incremental settings when compared to its non-incremental counterparts. Finally, our analyses show that our streaming model learns to wait the most when it encounters an interregnum or reparandum, and the least for fluent or edit terms.
Although disfluency detection itself is a well-studied task, only a handful of past work has explored disfluency detection in an online setting that is, consuming the input a single token at a time and outputting predictions on the partial input as early as possible. Among the neural approaches, Hough and Schlangen (2015) were the first to demonstrate competitive performance of recurrent neural networks (RNNs) on incremental disfluency detection by applying an Elman RNN paired with a Markov decoder that jointly optimized the probability of the output tag over the past inputs and outputs. Hough and Schlangen (2017) built upon this by jointly training LSTMs on both utterance segmentation and disfluency detection, demonstrating that jointly training on the two tasks yielded higher accuracy and lower latency on both tasks than training on either task alone. This was followed by a number of other works that also suc
Figure 2: The architecture of our streaming BERT model. The disfluency classification head outputs predictions of fluent  $(f)$  or disfluent  $(d)$  for each token, whereas the wait classification head outputs predictions of predict  $(p)$  or wait  $(w)$  for each token. Given a partial input, we find the first token with a wait prediction and output only the tokens before it with fluent predictions.
cessfully paired incremental disfluency detection with other tasks, such as language modeling (Shalyminov et al., 2018) and POS tagging Rohanian and Hough (2020).
More recently, large pre-trained transformer architectures have demonstrated incredible success on sequence labeling tasks (Vaswani et al., 2017). Although the original transformer architecture was not designed for streaming input, Chen et al. (2020) proposed the controllable time-delay transformer instead, which combines a fast decoding strategy with a modified self-attention mechanism that attends only to future inputs in a fixed lookahead window.
The closest work to ours is that of Rohanian and Hough (2021), in which the authors fine-tuned a pre-trained  $\mathrm{BERT}_{\mathrm{LARGE}}$  model via add-  $M$  training, which feeds the model successive prefixes of lengths  $N + M, N + 2M, \dots$  for each full-length training example. Their best-performing model also made use of a prophecy decoding strategy, in which a GPT-2 (Radford et al., 2019) model predicted the missing right context of each partial input. The BERT model then made its predictions based on the complete extrapolated sequence, POS tags, and word timings. Unlike their work, we aim to train a more lightweight small vocabulary  $12 \times 128$  BERT model that is more suitable for on-device settings, does not require a separate prophecy generation model, and uses only disfluency-annotated training data. We also train our model on successive prefixes of the input (with  $N = 1, M = 1$ ) but modify both the architecture and training objective in order to balance the competing objectives of accuracy and latency.
3 Training a Streaming BERT Model
In this section we describe the architectural changes, new training objective, and training scheme that we use to adapt a (non-incremental) BERT model for streaming sequence tagging tasks. Specifically, we modify the model to enable it to either immediately produce a prediction for a given token or to decide to wait for further input. These changes include a novel training objective that balances the cost between accuracy and latency - preventing the model from the extremes of either relying too much on waiting for further input or on speedily making predictions at the cost of accuracy. We train this model using a restart-incremental training procedure described in Section 3.2.
Streaming settings force models to make trade-offs between accuracy and latency. More accurate predictions can be obtained by providing longer right context at the cost of incurring additional latency. However, since most tokens are not disfluent, a model may not require the right context in order to accurately classify fluent tokens. Rather than using a fixed lookahead window, we train a BERT model to jointly classify tokens and simultaneously choose the lookahead window size dynamically at each token.
Our proposed model architecture consists of a pre-trained BERT model with two separate token classification heads added on top, as shown in Figure 2. Each classification head consists of a linear layer applied to the hidden layer outputs of the BERT model. The first classification head, the disfluency classifier, is trained to classify whether
each token is disfluent or not. The second classification head (the wait classifier) is trained to classify whether the model should wait for further input (temporarily abstain from predicting) or immediately output a prediction for the given token. In effect, the wait classifier decides how large of a lookahead window the model needs to make its prediction on the current input. At inference time, we only output tokens that lie to the left of the first token for which the model outputs a wait prediction and that are predicted to be fluent. This avoids potentially producing disjoint output in the case where the model produces predictions of wait followed by predict, making the output more clear for the user's display.
We also adapt the training objective such that it accounts for both the accuracy and latency of the model's outputs on each successive prefix. Let  $(x,y)$  be the pair of input sequence and target output sequence with prefixes  $x_{1},x_{2},\dots ,x_{|x|}$  and  $y_{1},y_{2},\dots ,y_{|x|}$ , respectively where  $|x|$  is the length of the full sentence. We also denote  $f(x)$  as the output logits of the disfluency classifier,  $g(x)$  as the output logits of the wait classifier,  $\sigma (\cdot)$  as the softmax function, and  $H(\cdot ,\cdot)$  as the cross-entropy loss. Then the traditional cross-entropy loss on the full input and target sequences is
$$
\ell_ {\text {F U L L}} (x, y) = H (\sigma (f (x)), y) \tag {1}
$$
However, for each prefix we wish to only compute the cross-entropy loss on the tokens to the left of the first token for which the model outputs a wait prediction. To accomplish this, we devise a binary mask that zeros out the loss on the tokens to the right of and including the first wait prediction:
$$
\mathbf {m} \left(\sigma \left(g \left(x _ {i}\right)\right) = \left(m _ {1}, \dots , m _ {\left| x _ {i} \right|}\right)\right) \tag {2}
$$
$$
m _ {j} = \left\{ \begin{array}{l l} 1 & \text {i f} j <   k \\ 0 & \text {o t h e r w i s e} \end{array} \right. \tag {3}
$$
$$
k = \min  \left\{j \mid \sigma \left(g \left(x _ {i}\right) _ {j}\right) > 0. 5 \right\} \tag {4}
$$
where  $g(x_{i})_{j}$  is the  $j$ -th element of vector  $g(x_{i})$ . We then apply this mask to the cross-entropy loss for each prefix of example  $x$  to obtain prefix loss:
$$
\ell_ {\mathrm {P R E F I X}} (x, y) = \sum_ {i = 1} ^ {| x | - 1} \mathbf {m} (\sigma (g (x _ {i}))) \circ H (\sigma (f (x _ {i})), y _ {i}) \tag {5}
$$
where we abuse notation here by denoting  $H(\sigma(f(x_i)), y_i)$  as the vector for which the  $j$ -th element corresponds to the cross-entropy of  $(\sigma(f(x_i))_j, y_{i,j})$  and  $\circ$  is element-wise multiplication. Lastly, we define a latency cost that scales with both the probability of abstaining from classifying the  $j$ -th token in the  $i$ -th prefix  $(\sigma(g(x_i)_j))$  and with the expected wait time, as measured by number of tokens, incurred by abstaining starting from token  $j$  in prefix  $x_i$ :
$$
\ell_ {\text {L A T E N C Y}} (x) = \sum_ {i = 1} ^ {| x | - 1} \sum_ {j = 1} ^ {i} (i - j) \sigma \left(g \left(x _ {i}\right) _ {j}\right) \tag {6}
$$
Putting these together, the total loss for a single example  $(x,y)$  is:
$$
\begin{array}{l} \ell (x, y) = \ell_ {\text {F U L L}} (x, y) + \gamma \ell_ {\text {P R E F I X}} (x, y) \\ + \lambda \ell_ {\text {L A T E N C Y}} (x) \tag {7} \\ \end{array}
$$
with hyperparameters  $\gamma$  and  $\lambda$  controlling the relative strengths of the prefix and latency costs, respectively. We also include the cross-entropy loss on the full sequences  $(\ell_{\mathrm{FULL}})$  in addition to the prefix losses  $(\ell_{\mathrm{PREFIX}})$  because we wish for the model to maintain its ability to make predictions on full sequences. Since  $g(x)$  does not appear anywhere in  $\ell_{\mathrm{FULL}}$ , the model is effectively forced to make predictions once it receives the full utterance.
Similarly, the  $\ell_{\mathrm{LATENCY}}$  term is essential because without it, the model could achieve minimal loss by always waiting (e.g.  $\sigma(g(x_i)_j) = 1$  for all prefixes  $i$  and time steps  $j$ ), and only learning to classify disfluent tokens after receiving the full sequence. This is equivalent to the non-incremental classification loss. If we instead set  $\sigma(g(x_i)_j) = 0$  for all  $i, j$  (the case where the model never waits), the resulting loss is equivalent to the learning objective for strongly incremental training (see Section 3.2). In essence, our training objective is a generalization of the strongly incremental objective.
3.2 Restart-Incremental Training
Although BERT models are typically fine-tuned using complete pre-segmented sequences, an incremental model must process partial inputs at inference time, resulting in a distributional shift between the complete utterances typically seen in training datasets and the partial utterances seen at inference time. A simple solution is to fine-tune BERT both on complete and partial inputs, a training scheme known as restart incrementality (Ka-hardipraja et al., 2021). By providing successively
extended prefixes of a given utterance to the model and computing the loss on the model outputs for each prefix, we can mimic the streaming data that the model would encounter in real time. In all of our experiments, each successive prefix adds a single word to the previous prefix, a setting known as strongly incremental (Shalyminov et al., 2018). Although this approach requires re-computation of the model outputs for each successive prefix, this also enables the model to correct its previous predictions, or to switch between waiting and predicting when it receives helpful right context. Incorporating prefixes during training in incremental disfluency detection has been previously explored by Rohanian and Hough (2021). This serves as a strong baseline in our experiments.
We fine-tune all models on the Switchboard dataset (Godfrey et al., 1992), a transcribed English multispeaker conversational corpus that is commonly used for ASR research. We specifically use the version from the Linguistic Data Corpus's Treebank-3 (Marcus et al., 1999) distribution, which additionally contains disfluency annotations and a standard train/dev/test split (Charniak and Johnson, 2001). We follow Rocholl et al. (2021), training our models to classify both the reparanda and interregna as disfluent for future removal in a final post-processed transcript.
All of our experiments use small distilled BERT models, specifically a small vocabulary BERT model (Zhao et al., 2021)  $(\mathrm{BERT}_{\mathrm{SV}})$  with 12 hidden layers of size 128 that is pre-trained on English Wikipedia and BookCorpus (Zhu et al., 2015). The details of our hyperparameter tuning can be found in the Appendix (Section A.2).
We use small models for two reasons: 1) fine-tuning a model on all given prefixes of each training example is resource intensive, and 2) many streaming natural language understanding applications run entirely on mobile devices which precludes the use of large models. Previous work on small non-incremental BERT-based models used for disfluency detection (Rocholl et al., 2021) showed significant improvement in memory and latency without compromising task performance. The core  $\mathrm{BERT}_{\mathrm{SV}}$  model is a distilled version of  $\mathrm{BERT}_{\mathrm{BASE}}$  with smaller vocabulary and reduced hidden layer
dimensions (Zhao et al., 2021). Due to its smaller vocabulary size (5K versus 30K tokens), the model has only about 3.1M parameters, as compared to  $\mathrm{BERT}_{\mathrm{BASE}}$ ’s approximately 108.9M parameters, achieving around  $80\%$  latency reduction.
In order to isolate the effects of training with restart incrementality (Section 3.2) versus the improvements derived directly from incorporating our new training objective, we also evaluate two other models: 1) a non-incremental  $\mathrm{BERT}_{\mathrm{SV}}$  model trained in the usual way, on full sequences; and 2) a  $\mathrm{BERT}_{\mathrm{SV}}$  model trained with restart incrementality - i.e., on all prefixes of every training example (which we will refer to as "all prefixes" in following tables). Setup (1) is equivalent to ablating both  $\ell_{\mathrm{PREFIX}}$  and  $\ell_{\mathrm{LATENCY}}$  whereas setup (2) is equivalent to ablating only  $\ell_{\mathrm{LATENCY}}$ . We do not ablate  $\ell_{\mathrm{PREFIX}}$  in isolation since this leaves only the  $\ell_{\mathrm{FULL}}$  and  $\ell_{\mathrm{LATENCY}}$  terms, and there does not exist a meaningful measure of latency when the model never needs to wait for more input (since it is always given the full utterance as input). For each of these baseline models we also follow Rohanian and Hough (2021) and Kahardipraja et al. (2021) by evaluating with different fixed lookahead (LA) window sizes of LA = 0, 1, 2.
4.2 Incremental Evaluation
Accuracy alone is not a sufficient measure of success to robustly evaluate a streaming model. Since a streaming model is meant to operate in real time, it should return output as soon as possible after it receives new input. As such, we also need to evaluate it with respect to latency – i.e. the number of new tokens a model must consume before producing a prediction for the current token. Furthermore, streaming models are often designed to be capable of retroactively changing their predictions on previous tokens as new input arrives. This introduces the risk of output “jitter” or “flicker,” where the output changes dramatically as new input is consumed, necessitating evaluation of stability. To capture all these important dimensions of streaming model performance, we evaluate the models using the following diachronic metrics (with formulas and further details in the Appendix):
- Streaming  $F_{1}$ : An accuracy metric scored in the same way as the typical  $F_{1}$  score, albeit we score the predictions for a single token over the course of multiple time steps separately as if they were predictions for separate tokens.
Model	Training Scheme	Incremental	Final
F1↑	P↑	R↑	EO↓	TTD↓	AWT↓	F1↑
BERTSV	Full sequences	0.76	0.74	0.78	0.31	1.46	0.00	0.89
BERTSV	All prefixes	0.76	0.73	0.78	0.32	1.37	0.00	0.89
Streaming BERTSV	All prefixes	0.83	0.92	0.75	0.09	2.32	0.21	0.88
Models with lookahead ≥ 1
BERTSV (LA = 1)	Full sequences	0.83	0.85	0.80	0.10	2.41	1.00	0.89
BERTSV (LA = 2)	Full sequences	0.85	0.89	0.82	0.05	3.06	2.00	0.89
BERTSV (LA = 1)	All prefixes	0.82	0.85	0.80	0.12	2.33	1.00	0.89
BERTSV (LA = 2)	All prefixes	0.85	0.89	0.82	0.06	3.01	2.00	0.89
Table 1: Comparison of incremental performance on the Switchboard validation set of non-incremental small-vocab BERT models  $(\mathrm{BERT}_{\mathrm{SV}})$  against that of a streaming small-vocab BERT model (streaming  $\mathrm{BERT}_{\mathrm{SV}}$ ). In the lower half of the table we also list the evaluation results of non-incremental  $\mathrm{BERT}_{\mathrm{SV}}$  models with fixed lookahead (LA) window sizes of 1 and 2 tokens. Note that for the non-incremental models the lookahead window size is equivalent to the average waiting time (AWT). The arrows near each metric represent the desirable direction of the result:  $\uparrow$  means the higher the performance the better and  $\downarrow$  is the reverse.
- Edit Overhead (EO) (Buß and Schlangen, 2011): A stability metric that measures the average number of unnecessary edits, normalized by utterance length.
- Time-to-detection (TTD) (Hough and Schlangen, 2017): A latency metric that is only computed on disfluent tokens that are classified correctly. It is the average amount of time (in number of tokens consumed) that the model requires before first detecting a disfluency. As mentioned earlier, we include both reparanda and interregna as disfluencies.
- Average waiting time (AWT): The average amount of time (in number of tokens consumed) that the model waits for further input before making a prediction on a given token. For models with a fixed lookahead window, this is equivalent to the lookahead window size. For the streaming model, this is equivalent to the average lookahead window size.
- First time to detection (FTD) (Zwarts et al., 2010; Rohanian and Hough, 2021): Similar to the TDD metric described above with the main difference being that the latency (in number of words) is calculated starting from the onset of a gold standard repair.