SlowGuess/ACL_2020_OCR huggingface dataset:
Paria Jamshid Lou<sup>1</sup> and Mark Johnson<sup>2,1</sup>
$^{1}$ Department of Computing, Macquarie University
$^{2}$ Oracle Digital Assistant, Oracle Corporation
<sup>1</sup>paria.jamshid-lou@hdr.mq.edu.au
$^{2}$ mark.mj.johnson@oracle.com
Self-attentive neural syntactic parsers using contextualized word embeddings (e.g. ELMo or BERT) currently produce state-of-the-art results in joint parsing and disfluency detection in speech transcripts. Since the contextualized word embeddings are pre-trained on a large amount of unlabeled data, using additional unlabeled data to train a neural model might seem redundant. However, we show that self-training — a semi-supervised technique for incorporating unlabeled data — sets a new state-of-the-art for the self-attentive parser on disfluency detection, demonstrating that self-training provides benefits orthogonal to the pre-trained contextualized word representations. We also show that assembling self-trained parsers provides further gains for disfluency detection.
Speech introduces challenges that do not appear in written text, such as the presence of disfluencies. Disfluency refers to any interruptions in the normal flow of speech, including false starts, corrections, repetitions and filled pauses. Shriberg (1994) defines three distinct parts of a speech disfluency, referred to as the reparandum, the interregnum and the repair. As illustrated in the example below, the reparandum The first kind of invasion of is the part of the utterance that is replaced or repaired, the interregnum uh I mean (which consists of a filled pause uh and a discourse marker I mean) is an optional part of the disfluency, and the repair the first type of privacy replaces the reparandum. The fluent version is obtained by removing the reparandum and the interregnum.
This paper will focus on joint disfluency detection and constituency parsing of transcribed speech. In the Switchboard treebank corpus (Godfrey and Holliman, 1993; Marcus et al., 1999), which is a standard corpus for parsing studies on conversational speech, the reparanda, filled pauses and discourse markers are dominated by EDITED, INTJ and PRN nodes, respectively (see Figure 1). Filled pauses and discourse markers belong to a finite set of words and phrases, so INTJ and PRN nodes are trivial to detect (Johnson and Charniak, 2004). Detecting EDITED nodes, however, is challenging and is the main focus of disfluency detection models.
Figure 1: A parse tree from the Switchboard corpus, where reparandum The first kind of invasion of, filled pause uh and discourse marker I mean are dominated by EDITED, INTJ and PRN nodes.
Jamshid Lou et al. (2019) showed that a self-attentive constituency parser achieves state-of-the-art results for joint parsing and disfluency detection. They observed that because the Switchboard
3754
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3754-3763 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
trees include both syntactic constituency nodes and EDITED nodes that indicate disfluency, training a parser to predict the Switchboard trees can be regarded as multi-task learning (where the tasks are syntactic parsing and identifying disfluencies). In this paper, we extend the multi-task learning in Jamshid Lou et al. (2019) to explore the impact of self-training (McClosky et al., 2006) and assembling (Kitaev et al., 2019) on the performance of the self-attentive parser. We aim to answer two questions about the state-of-the-art self-attentive parser:
- Does self-training improve the performance of the self-attentive parser on disfluency detection? Self-training is a semi-supervised technique for incorporating unlabeled data into a new model, where an existing model trained on manually labeled (i.e. gold) data is used to label unlabeled data. The automatically (i.e. silver) labeled data are treated as truth and combined with the gold labeled data to re-train a new model (McClosky et al., 2006; Choe and Charniak, 2016). Since neural models use rich representations of language pre-trained on a large amount of unlabeled data (Peters et al., 2018; Devlin et al., 2019), we might expect that self-training adds no new information to the self-attentive parser. Surprisingly, however, we find that self-training improves disfluency detection f-score of the BERT-based self-attentive parser, demonstrating that self-training provides benefits orthogonal to the pre-trained contextualized embeddings.
- Does ensembling improve disfluency detection in speech transcripts? Ensembling is a commonly used technique for improving parsing where scores of multiple instances of the same model trained on the same or different data are combined at inference time (Dyer et al., 2016; Fried et al., 2017; Kitaev et al., 2019). We expect ensembling parsers to improve the performance of the model on disfluency detection, too. We show ensembling four self-trained parsers (using different BERT word representations) via averaging their span label scores increases disfluency detection f-score in comparison with a single self-trained parser.
Parsing speech transcripts is challenging for conventional syntactic parsers, mainly due to the presence of disfluencies. In disfluent sentences, the relation between reparandum and repair is different from other words in the sentence. The repair is usually a "rough copy" of the reparandum, using the same or similar words in roughly the same word order<sup>1</sup> (Charniak and Johnson, 2001). Designed to capture tree-like structures, conventional syntactic parsers fail to detect "rough copies" which are strong indicators of disfluency. Moreover, the reparandum and repair often do not form a syntactic phrase, which makes detecting the reparandum even harder. For these reasons, specialized disfluency detection models were developed to remove disfluencies prior to parsing (Charniak and Johnson, 2001; Kahn et al., 2005; Lease and Johnson, 2006) or special mechanisms were added to parsers to handle disfluencies (Rasooli and Tetreault, 2013; Honnibal and Johnson, 2014; Yoshikawa et al., 2016). Conventional parsing based models can use the syntactic location of the disfluency as a feature in a reranker (Johnson et al., 2004). A similar gain can be achieved in neural models by training a joint parsing and disfluency detection model. In this multi-task learning setting, syntactic information helps the neural model detect disfluencies more accurately (Jamshid Lou et al., 2019).
State-of-the-art results for disfluency detection have been reported for Transformer models using contextualized embeddings (e.g. ELMo and BERT) (Jamshid Lou et al., 2019; Tran et al., 2019; Dong et al., 2019). The self-attention mechanism of the Transformer is apparently effective for capturing "rough copy" dependencies between words. A recent study shows that prosody slightly improves the parsing performance of the self-attentive model over the text-only model, especially in long sentences (Tran et al., 2019). In this paper, we use a self-attentive model for joint disfluency detection and constituency parsing.
Disfluency detection models are usually trained and evaluated on the Switchboard corpus. Switchboard is the largest disfluency annotated dataset. However, only  $5.9\%$  of the words in the Switchboard are disfluent (Charniak and Johnson, 2001).
<sup>1</sup>For example in Figure 1, the reparandum The first kind of invasion of and the repair the first type of privacy are "rough copies" of each other.
To mitigate the scarcity of labeled data, some studies have leveraged additional data by using: (i) contextualized embeddings pre-trained on enormous amount of unlabeled data (Jamshid Lou et al., 2019; Tran et al., 2019; Bach and Huang, 2019) and (ii) synthetic data generated by adding noise in the form of disfluencies to fluent sentences (e.g. repeating, deleting or inserting words in a sentence) (Wang et al., 2018; Bach and Huang, 2019; Dong et al., 2019). By contrast, this paper focuses on self-training, which is a simple semi-supervised technique that has been effective in different NLP tasks, including parsing (McClosky et al., 2006; Clark et al., 2018; Droganova et al., 2018). To our best knowledge, this is the first work that investigates self-training a neural disfluency detection model.
Another technique commonly used for improving parsing is ensembling. Ensembling is a model combination method, where scores of multiple models (they can be the same or different models, trained on the same or different data, with different random initializations) are combined in some way (Dyer et al., 2016; Choe and Charniak, 2016; Fried et al., 2017). The state-of-the-art for parsing written text is an ensemble of four BERT-based self-attentive parsers, where the parsers are combined by averaging their span label scores (Kitaev et al., 2019). While ensembling is widely used in parsing, it has not been investigated for disfluency detection. In this paper, we also explore the impact of ensembling several parsing based disfluency detection models on disfluency detection performance.
Following Jamshid Lou et al. (2019), we use a self-attentive constituency parser for joint disfluency detection and syntactic parsing<sup>2</sup>. The parsing model is based on the architecture introduced by Kitaev and Klein (2018), which is state-of-the-art for (i) parsing written texts (Kitaev et al., 2019; Fried et al., 2019), (ii) parsing transcribed speech (Tran et al., 2019), and (iii) joint parsing and disfluency detection (Jamshid Lou et al., 2019).
The self-attentive parser assigns a score  $s(T)$  to each tree  $T$  by calculating the sum of the potentials
on its labeled constituent spans:
$$
s (T) = \sum_ {(i, j, l) \in T} s (i, j, l) \tag {1}
$$
where  $s(i,j,l)$  is the score of a constituent beginning at string position  $i$  ending at position  $j$  with label  $l$ . The input to the parser is a sequence of vectors corresponding to the sequence of words in a sentence followed by one or more self-attention layers. For each span  $(i,j)$ , a hidden vector  $h_{ij}$  is constructed by subtracting the representations of the start and end of the span. A span classifier, including two fully connected layers followed by a non-linearity, assigns labeling scores  $s(i,j,.)$  to each span. Then, the highest scoring parse tree is found for a given sentence as follows:
$$
\hat {T} = \underset {T} {\operatorname {a r g m a x}} s (T) \tag {2}
$$
using a modified CYK algorithm. The parser introduced in Kitaev and Klein (2018) relies on an external POS tagger to predict preterminal labels, but because the parser's accuracy does not decrease when no external POS tagger is used, we use their parser here without an external POS tagger (hence, all the preterminal labels are UNK). For more details, see Kitaev and Klein (2018).
3.1 Contextualized Embeddings
We incorporate BERT (Devlin et al., 2019) in our self-attentive parser by fine-tuning the parameters as part of the training process. Following Kitaev et al. (2019), we apply a learned projection matrix on the output of BERT to project the vectors to our desired dimensionality. The representations are then fed into the parser. BERT learns the representations for sub-word units, so to extract the word representations, we consider the representations of the last sub-word unit for each word in the sentence (Kitaev et al., 2019).
3.2 Self-Training
We train the self-attentive parser on the Penn Treebank-3 Switchboard corpus which contains gold disfluency labeled parse trees (Godfrey and Holliman, 1993; Marcus et al., 1999). Using the trained model, we parse unlabeled data and add the silver parse trees to the gold Switchboard training data and re-train the self-attentive parser using the enlarged training set. The unlabeled data we use include Fisher Speech Transcripts Part 1 (Cieri
The code is available at: https://github.com/pariajm/joint-disfluency-detection-and-parsing
et al., 2004) and Part 2 (Cieri et al., 2005). Table 1 summarizes the different datasets used to train the self-attentive parser.
Dataset	Labels	# Sents	# Words
SWB	gold	98k	733k
Fisher	silver	835k	14m
Table 1: Summary of the datasets used to train the self-attentive parser.
Following Charniak and Johnson (2001), we split the Switchboard into training, dev and test sets as follows: training data consists of the sw[23].*.mrg files, dev data consists of the sw4[5-9].*.mrg files and test data consists of the sw4[0-1].*.mrg files. All partial words $^3$  and punctuations are removed from the data, as they are not available in realistic ASR applications (Johnson and Charniak, 2004).
Our baseline is the self-attentive parser trained on the gold Switchboard corpus with BERT word representations. The BERT-based parser is the current state-of-the-art, providing a very strong baseline for our work. We trained different versions of the baseline parser using four different BERT models, namely  $\mathrm{BERT}_{\mathrm{BASE}}$  [cased|uncased] and  $\mathrm{BERT}_{\mathrm{LARGE}}$  [cased|uncased], and then selected the best model i.e.  $\mathrm{BERT}_{\mathrm{BASE}}$  [cased] on the Switchboard dev set. We also tuned the hyperparameters by optimizing for performance on parsing EDITED nodes  $F(S_{E})$ . Preliminary experiments on the Switchboard dev set showed that the hyperparameters given by Kitaev et al. (2019) perform well; therefore, this is what we used here. Since random seeds lead to different results, in this paper we report average scores across 5 runs of each model initialized with different random seeds.
We evaluate the self-attentive parser in terms of parsing accuracy, as well as disfluency detection. Since certain words are identified as EDITED in the parse tree, we can measure how well a parser
classifies words as EDITED. We can also evaluate how accurately the parser can identify all disfluency words, i.e., the words dominated by EDITED, INTJ or PRN nodes. Therefore, we report precision (P), recall (R) and  $f$ -score (F) for both constituent spans (S) and word positions (W), where each word position is treated as labeled by all the constituents containing that word. We also report the result for subsets of constituent spans and word positions: (i)  $S_E$ , the set of constituent spans labeled EDITED, (ii)  $W_E$ , the set of word positions dominated by one or more EDITED nodes, and (iii)  $W_{EIP}$ , the set of word positions dominated by one or more EDITED, INTJ or PRN nodes. For more details, see Jamshid Lou et al. (2019).
4.3 Varying Amount of Silver Training Data
To find the optimal proportion of additional silver training data, we select  $n$  percent (ranging from  $10\%$  to  $90\%$ ) of the training data in each minibatch from silver parse trees and the rest from the gold ones. This has the same effect as reweighting the main gold corpus as in McClosky et al. (2006). The results for using different proportions of the silver parse trees are presented in Figure 2. The BERT-based parser self-trained with  $40\%$  silver Fisher trees and  $60\%$  gold Switchboard trees is our best model. In other words, for a batch size of 30, in each mini-batch 12 parse trees come from the silver Fisher data and 18 parse trees from the gold Switchboard. All self-training results in this paper use this proportion of gold and silver parse trees.
Figure 2: EDITED node f-score  $F(S_{E})$  of the BERT-based self-attentive parser as a function of percentages of training data in each mini-batch sourced from silver Fisher trees.
3Words tagged as "XX" or words ending in "-
4.4 Does self-training improve the performance of the self-attentive parser?
Tables 2 and 3 compare the baseline and the self-trained parser in terms of parsing and disfluency detection. The parser self-trained on the silver Fisher data increases parsing and disfluency detection performance, indicating the BERT-based model benefits from additional silver labeled data. Self-training is especially effective for recognizing EDITED disfluency nodes (1.5% increase in f-score). Only 5.9% of the words in the Switchboard are disfluent, and BERT is only trained on fluent texts such as books and Wikipedia, so the baseline parser may be starved of disfluent training examples. As a result, self-training on a corpus of conversational speech may compensate for the scarcity of disfluent gold data. To explore this, we tried self-training on a wide variety of fluent clean datasets, including Gigaword 5 (which is an unlabelled newswire corpus) and WSJ and Brown (which include gold parse trees of written text), but the performance did not improve significantly. This suggests that the parser benefits more from additional in-domain (i.e. conversational) silver data than additional out-of-domain (i.e. written) silver/gold data. Moreover, if we learn the embeddings as part of training instead of using pre-trained BERT, EDITED word f-score would drop from 90.9% to 86.4% and self-training on Fisher leads to little improvement (0.2% increase in EDITED word f-score compared to 1.5% improvement when using BERT). This suggests that self-training works well when the baseline model is powerful enough to predict accurate silver labels.
Parsing	F(SE)	F(SEIP)	F(S)
Baseline	89.2	95.6	93.5
Self-trained	90.7	96.2	93.9
To further investigate the influence of selftraining on disfluency detection, we randomly select 100 sentences containing disfluencies from
Table 2: Parse f-score for EDITED node  $F(S_{E})$ , for EDITED, INTJ and PRN nodes  $F(S_{EIP})$  and for all constituent spans  $F(S)$  on the Switchboard dev set for the baseline parser and the parser trained on the silver Fisher data.
Disfluency	F(WE)	F(WEIP)
Baseline	90.9	95.3
Self-trained	92.4	96.0
the Switchboard dev set. We categorize disfluencies into repetition, correction and restart according to Shriberg's (1994) typology of speech repairs. Repetitions are repairs where the reparandum and repair portions of the disfluency are identical, while corrections are where the reparandum and repairs differ (which are much harder to detect). Restarts are where the speaker abandons a sentence and starts a new one (i.e. the repair is empty). As Table 4 shows, the self-trained parser outperforms the baseline in detecting all types of disfluency. It especially has a better performance at detecting corrections and restarts which are more challenging types of disfluency in comparison with repetitions.
Table 3: EDITED word f-score  $F(W_{E})$ , EDITED, INTJ and PRN word f-score  $F(W_{EIP})$  on the Switchboard dev set for the baseline parser and the parser trained on the silver Fisher data.
Model	Rep.	Cor.	Res.	All
Baseline	97.0	80.6	82.0	89.2
Self-trained	97.3	88.6	87.8	92.9
Table 4: EDITED word f-score  $F(W_{\mathrm{E}})$  for different types of disfluency on a subset of the Switchboard dev set containing 158 disfluent structures — including 90 repetitions (Rep.), 54 corrections (Cor.) and 14 restarts (Res.).
4.5 Does ensembling parsers improve disfluency detection?
We investigate the impact of ensembling on the performance of the self-attentive parser, where we combine parsers by averaging their span label scores as follows:
$$
s _ {\text {e n s e m b l e}} (i, j, l) = \frac {1}{4} \sum_ {n = 1} ^ {4} s _ {n} (i, j, l) \tag {3}
$$
We tried different ensembling of parsers and the best result was achieved when we trained the baseline parser four times using four BERT word rep
#	Model	EDITED Disfluency Labels
1	Gold	if if you call the any eight hundred number if you you can call up any eight hundred number
Baseline	if if you call the any eight hundred number if you you can call up any eight hundred number
Self-trained	if if you call the any eight hundred number if you you can call up any eight hundred number
2	Gold	she was going to get picked up she was going to pick him up because she only ...
Baseline	she was going to get picked up she was going to pick him up because she only ...
Self-trained	she was going to get picked up she was going to pick him up because she only ...
3	Gold	It goes back to you know what right what can society impose on people
Baseline	It goes back to you know what right what can society impose on people
Self-trained	It goes back to you know what right what can society impose on people
4	Gold	and the money they do have they're not they do not use it wisely
Baseline	and the money they do have they're not they do not use it wisely
Self-trained	and the money they do have they're not they do not use it wisely
5	Gold	For two years we didn't and we which was a kind of stupid
Baseline	For two years we didn't and we which was a kind of stupid
Self-trained	For two years we didn't and we which was a kind of stupid
6	Gold	We we couldn't survive in a in a juror in a trial system without a jury
Baseline	We we couldn't survive in a in a juror in a trial system without a jury
Self-trained	We we couldn't survive in a in a juror in a trial system without a jury
7	Gold	... I think it's like ninety-nine point ninety-nine think it is
Baseline	... I think it's like ninety-nine point ninety-nine think it is
Self-trained	... I think it's like ninety-nine point ninety-nine think it is
8	Gold	Do you think for a big or a little place
Baseline	Do you think for a big or a little place
Self-trained	Do you think for a big or a little place
resentations, namely  $\mathrm{BERT}_{\mathrm{BASE}}$  [cased|uncased] and  $\mathrm{BERT}_{\mathrm{LARGE}}$  [cased|uncased], and combined the results at inference time (Kitaev et al., 2019). The ensembled models not only reflect variations of different pre-trained representations but also the randomness in initialization of the models. As shown in Table 6, ensembling and self-training both improve the performance of the baseline single model on parsing and detecting EDITED disfluency nodes. Self-training is more effective than ensembling, especially for EDITED node detection. The best results are reported for ensembling the best of the self-trained parsers for each of different BERT models from the 5 random restarts<sup>4</sup>.
Table 5: Some examples from the Switchboard dev set and corresponding EDITED disfluency labels given by the baseline and the best self-trained parser, as well as the gold (i.e. correct) labels. Green (and italic) words indicate correctly labeled disfluent words and orange (and underlined) words represent fluent words which are incorrectly labeled as disfluencies.
Model	F(SE)	F(WE)	F(WEIP)
Baseline (single)	89.2	90.9	95.3
Baseline (ensemble)	90.3	91.1	95.6
Self-trained (single)	90.7	92.4	96.0
Self-trained (ensemble)	90.9	92.8	96.4
Table 6: Parse f-score for EDITED node  $F(S_{E})$ , EDITED word f-score  $F(W_{E})$  and EDITED, INTJ and PRN word f-score  $F(W_{EIP})$  for different models on the Switchboard dev set. "single" = single parser and "ensemble" = ensemble of 4 parsers.
4We also tried ensembling all 20 versions of the self-trained parser initialized with different random seeds. The
results were  $0.1\%$  worse than the ensemble of the four best self-trained parsers.
(b) Self-trained
Figure 3: A sentence from the Switchboard dev set parsed by the baseline model (left) and by the self-trained model (right). The parse tree obtained by the self-trained model is the same as the gold parse tree.
We compare the performance of our best model with previous work on the Switchboard test set. As demonstrated in Table 7, our model outperforms prior work in parsing. The parsing result for our model is higher than Tran et al. (2018) which utilizes prosodic cues, as well as text based features.
Parsing (S)	P	R	F
Tran et al. (2018)	-	-	87.9
Tran et al. (2018)*	-	-	88.5
Jamshid Lou et al. (2019)	92.4	92.9	92.7
Tran et al. (2019)	-	-	92.8
Tran et al. (2019)*	-	-	93.0
This work (single model)	93.2	93.8	93.5
This work (ensemble of 4)	93.6	94.2	93.9