SlowGuess/NAACL_2022_OCR huggingface dataset:
Ankita Pasadena\*2, Felix  $\mathbf{W}\mathbf{u}^{1}$  , Suwon Shon', Karen Livescu?, Kyu J. Han
$^{1}$ ASAPP  $^{2}$ Toyota Technological Institute at Chicago
Spoken language understanding (SLU) tasks involve mapping from speech signals to semantic labels. Given the complexity of such tasks, good performance is expected to require large labeled datasets, which are difficult to collect for each new task and domain. However, recent advances in self-supervised speech representations have made it feasible to consider learning SLU models with limited labeled data. In this work, we focus on low-resource spoken named entity recognition (NER) and address the question: Beyond self-supervised pre-training, how can we use external speech and/or text data that are not annotated for the task? We consider self-training, knowledge distillation, and transfer learning for end-to-end (E2E) and pipeline (speech recognition followed by text NER) approaches. We find that several of these approaches improve performance in resource-constrained settings beyond the benefits from pre-trained representations. Compared to prior work, we find relative improvements in F1 of up to  $16\%$ . While the best baseline model is a pipeline approach, the best performance using external data is ultimately achieved by an E2E model. We provide detailed comparisons and analyses, developing insights on, for example, the effects of leveraging external data on (i) different categories of NER errors and (ii) the switch in performance trends between pipeline and E2E models.
Named entity recognition (NER) is a popular task in natural language processing. It involves detecting the named entities and their categories from a text sequence. NER can be used to extract information from unstructured data, which can also be used as features for other tasks like question answering (Chen et al., 2017) and slot filling for task-oriented dialogues (Louvan and Magnini, 2018).
Figure 1: Improvements in spoken NER with 100 hours of external data of different types. "Pipeline" refers to approaches consisting of speech recognition followed by a text NER model; "E2E" refers to approaches that directly map from speech to NER-tagged text. The "Baseline" and "Text NER" numbers are from previously established baselines (Shon et al., 2022).
Thanks to pre-trained text representations such as BERT (Devlin et al., 2019), text-based NER has recently improved greatly (Wang et al., 2021b; Li et al., 2020b). Spoken NER, on the other hand, is not as well-studied. It has the added challenges of continuous-valued and longer input sequences and, at the same time, provides opportunities to take advantage of acoustic cues in the input. A recent study (Shon et al., 2022) shows that there is still  $10 - 20\%$  absolute degradation in the F1 scores of spoken NER models compared to text-based NER using gold transcripts (see Figure 1) despite using large pre-trained speech representation models. Closing this gap remains a critical challenge.
In this work, we study the potential benefits of using a variety of external data types: (a) plain speech audio, (b) plain text, (c) speech with transcripts, and (d) text-based NER data. We benchmark our findings against recently published baselines for NER on the VoxPopuli dataset of European Parliament speech recordings (Shon et al., 2022) and also introduce baselines of our own. We observe improvement from leveraging every type of external data. Our analysis also quantifies the pros and cons of the pipeline (speech recognition followed
* Work done during an internship at ASAPP.
Human Language Technologies, pages 724 - 737
by text NER) and end-to-end (E2E) approaches. The key improvements are summarized in Figure 1. Specific contributions include:
(i) Unlike previous work, we devote equal effort to improving both pipeline and E2E approaches.
(ii) We present experiments using various external data types and modeling approaches.
(iii) Overall, we obtain F1 improvements of up to  $16\%$  for the E2E model and  $6\%$  for the pipeline model over previously published baselines, setting a new state of the art for NER on this dataset.
(iv) We benchmark the advantage of self-supervised representations (SSR) over a baseline that uses standard spectrogram features. SSR gives relative improvements of  $36\% / 31\%$  for pipeline/E2E models, respectively. To our knowledge, prior work has not directly measured this improvement over competitive baselines tuned for the task.
(v) We establish that E2E models outperform pipeline approaches on this task, given access to external data, while the baseline models without the external data have the opposite relationship.
(vi) We provide a detailed analysis of model behavior, including differences in error types between pipeline and E2E approaches and the reasoning for the superiority of E2E over pipeline models when using external data but not in the baseline setting.
2.1 Spoken named entity recognition
Relatively little work has been conducted on spoken NER (Kim and Woodland, 2000; Sudoh et al., 2006; Parada et al., 2011; Ghannay et al., 2018; Caubrière et al., 2020; Yadav et al., 2020; Shon et al., 2022) as compared to the extensively studied task of NER on text (Nadeau and Sekine, 2007; Ratinov and Roth, 2009; Yadav and Bethard, 2018; Li et al., 2020a). While spoken NER is commonly done through a pipeline approach (Sudoh et al., 2006; Raymond, 2013; Jannet et al., 2015), there is rising interest in E2E approaches in the speech community (Ghannay et al., 2018; Caubrière et al., 2020; Yadav et al., 2020; Shon et al., 2022). These two approaches are depicted in Fig. 2.
An early E2E spoken NER model was introduced by Ghannay et al. (2018). The approach is based on the DeepSpeech2 (Amodei et al., 2016) architecture, with the addition of special characters for NER labels around the named entities in the transcription, and is trained with
character-level connectionist temporal classification (CTC) (Graves et al., 2006). Yadav et al. (2020) introduced an English speech NER dataset and proposed an E2E approach similar to Ghannay et al. (2018). They show that LM fusion improves the performance of the E2E approach. Caubrière et al. (2020) provided a detailed comparison between E2E and pipeline models; however, they focused on small RNN/CNN models and did not use state-of-the-art SSR models. All these approaches use at least 100 hours of annotated data.
These previous efforts have shown that E2E models can outperform pipeline approaches in a fully supervised setting. Borgholt et al. (2021) also made the same observation on a simplified NER task. However, these studies do not account for improvements in NLP from self-supervised text representations for their pipeline counterparts. Shon et al. (2022) introduced and worked with a low-resource NER corpus and showed that E2E models still do not rival pipeline approaches when state-of-the-art pre-trained models are used.
When using pre-trained representations, E2E models are at a disadvantage since the pipeline model also has access to a text model trained on  $>50\mathrm{GB}$  of text, in addition to the same speech representation model as E2E. This inspires us to study the benefits of using additional unlabeled data.
We choose to work with the NER-annotated VoxPopuli corpus (Wang et al., 2021a; Shon et al., 2022). VoxPopuli consists of naturally spoken speech, unlike Bastianelli et al. (2020), and is annotated manually, unlike Yadav et al. (2020) and Borgholt et al. (2021) who obtain ground-truth labels using text model predictions. The SLUE benchmark (Shon et al., 2022) is aimed at low-resource SLU and includes annotations for only 15 hours of data; this matches with the goals of our work, making it an ideal choice to benchmark our findings.
2.2 Self-supervised pre-training
There is a long history of using unsupervised pre-training in NLP to improve performance over limited-data supervised training on a broad range of tasks. SSRs have started to make an impact on speech tasks as well, with the first improvements seen in large-scale ASR with wav2vec (Schneider et al., 2019). More recently, improvements have been seen on more tasks with wav2vec 2.0 (Baevski et al., 2020) and other models (Yang et al., 2021).
Figure 2: High-level summary of approaches typically used to solve spoken and textual NER tasks. Optional LM decoding is applied in ASR and E2E-NER models.
However, it is not yet clear how universal these pre-trained representations are for speech tasks, particularly for semantic understanding tasks like NER. Some pre-trained models achieve impressive performance across a variety of tasks, including some understanding tasks (Yang et al., 2021) and analyses suggest that they contain at least some word meaning information (Pasad et al., 2021). However, these pre-trained models have not yet been tested on a broad range of challenging understanding tasks. We believe our work is the first to quantify the improvements from SSR, specifically on spoken NER.
2.3 Leveraging external data
Self-training (Scudder, 1965; Yarowsky, 1995; Riloff, 1996) is a popular approach to improve supervised models when some additional unannotated data is available. Self-training has been observed to improve ASR (Parthasarathi and Strom, 2019; Xu et al., 2021) and is also complementary to pre-training (Xu et al., 2021). To the best of our knowledge, this is the first work to introduce it to spoken NER while also studying its effects on both E2E and pipeline approaches.
Knowledge distillation is widely used in model compression research. In this approach, some intermediate output from a teacher model is used to train a smaller student model (Hinton et al., 2014). In the context of our work, the teacher and student networks are two different approaches to solving NER tasks, and the latter is trained on the final output tags of the former.
Transfer learning has been widely employed for SLU tasks (Lugosch et al., 2019; Jia et al., 2020), including E2E spoken NER (Ghannay et al., 2018; Caubrière et al., 2020). Automatic speech recog
nition (ASR) is a typically chosen task for pretraining a model before fine-tuning it for SLU. This choice is facilitated by the wider availability of transcribed speech than SLU annotations. Specifically for NER, ASR pre-training is expected to help since the accuracy of decoded texts can directly affect the final NER predictions.
Spoken NER involves detecting the entity phrases in a spoken utterance along with their tags. The annotations include the text transcripts for the audio and the entity phrases with their corresponding tags. Spoken NER, like any other SLU task, is typically tackled using one of two types of approaches: (i) Pipeline and (ii) End-to-end (E2E). As shown in Fig. 2, a pipeline approach decodes speech to text using ASR and then passes the decoded text through a text NER module, whereas an E2E system directly maps the input speech to the output task labels. Each approach has its own set of advantages and shortcomings. Pipeline systems can enjoy the individual advances from both the speech and the text research communities, whereas combining two modules increases inference time, and propagation of ASR errors can have unexpected detrimental effects on the text NER module performance. On the other hand, E2E models directly optimize a task-specific objective and tend to have faster inference. However, such models typically require a large amount of task-specific labeled data to perform well. This can be seen from previous papers on E2E NER (Yadav et al., 2020; Ghannay et al., 2018), where using at least 100 hours of labeled data is typical.
3.1 Baseline models
The baselines we use for E2E and pipeline models are taken from Shon et al. (2022). Similarly to previous work (Shon et al., 2022; Ghannay et al., 2018; Yadav et al., 2020), we formulate E2E NER as character-level prediction with tag-specific special characters delimiting entity phrases. For example, the phrases "irish" and "eu" are tagged as  $\mathrm{NORP}^1$  (\$) and  $\mathrm{GPE}^2$  (\%) respectively in "the  $\$$  irish  $J$  system works within a legal and regulatory policy directive framework dictated by the \% eu  $J$ "
The E2E NER and ASR modules are initialized with the wav2vec2.0 base (Baevski et al., 2020) pretrained speech representation, while the text NER module is pre-trained with DeBERTa base (He et al., 2021). These pre-trained models are then fine-tuned for ASR/NER after adding a linear layer on top of the final hidden-state output.
Since text transcripts are typically a part of the NER annotations, we can also train an NER model that uses the ground-truth text as input. This text NER model serves roughly as a topline and is further used in experiments with external data. The E2E NER and ASR models are trained with a character-level CTC objective. The text NER model is trained for token-level classification with cross-entropy loss.
It is expected that using self-supervised representations gives a significant boost in limited labeled data settings. In order to quantify the benefits of the pre-trained representations in our setting, we also report the performance of E2E and pipeline baselines that are trained from scratch, not utilizing any pre-trained models.
Similarly to previous work (Ghannay et al., 2018; Yadav et al., 2020), we evaluate performance using micro-averaged F1 scores on an unordered list of tuples of named entity phrase and tag pairs predicted for each sentence. An entity prediction is considered correct if both the entity text and the entity tag are correct.
Spoken NER introduces an added variability to the possible model errors due to speech-to-text conversion. We report word error rate (WER) to evaluate this aspect. WER is the word-level Levenshtein distance between the ground-truth text and the decoded text generated by the model. Additionally, to
get an idea of the errors made by the model specifically on named entities, we also evaluate NE ACC, the proportion of entity phrases correctly decoded in the speech-to-text conversion. An entity phrase is considered accurate only if all the words in the phrase are correctly decoded in the right order.
3.3 Utilizing external data
Next, we describe our approaches that use data external to the task-specific labeled data to improve both the pipeline and the E2E models for spoken NER. We consider four types of external data: (i) unlabeled speech  $(Un - Sp)$ , (ii) unlabeled text  $(Un - Txt)$ , (iii) transcribed speech  $(Sp - Txt)$ , and (iv) text-based NER data.
External data type	Method	Target model
Un-Sp	SelfTrain-ASR	ASR
Un-Txt	SelfTrain-txtNER	text NER
Sp-Txt	Pre-ASR	ASR
Table 1: Methods for using external data for pipeline models. For "SelfTrain" approaches, the labeling model is the same as the target model. The method for external transcribed data (Sp-Txt) is based on transfer learning and thus there is no labeling model. More details are provided in Sec. 3.3.
The majority of techniques we consider involve labeling the external data with a labeling model (typically one of the baseline models) to produce pseudo-labels. The target model is then trained on these generated pseudo-labels along with the original labeled NER data. Tables 1 and 2 present a detailed list of all methods we consider for improving pipeline and E2E models respectively. The methods we include use the first three kinds of external data listed above. The fourth kind, external text-based NER data, is used in experiments attempting to improve the text NER model; since it does not succeed (Sec. 5.2.1), this data source is not explored further for the pipeline and E2E models.
When the labeling model is the same as the target model, this is a well-established process called self-training (Scudder, 1965; Yarowsky, 1995; Riloff, 1996; Xu et al., 2020, 2021). In our setting, a word-level language model (LM) is used for decoding both the ASR and E2E NER models. Shon et al. (2022) observed that a LM consistently improves performance of all of the baseline models.
$^{1}$ NORP: Nationalities or religious or political groups
$^{2}$ GPE: Countries, cities, states
External data type	Method	Labeling model	Target model	LM for decoding
Un-Sp	SelfTrain-E2E	E2E-NER	E2E-NER	pLabel 4-gram
Distill-Pipeline	Pipeline-NER(after SelfTrain-ASR)	E2E-NER	pLabel 4-gram
Un-Txt	Distill-txtNER-lm	text NER	n/a	pLabel 4-gram
Sp-Txt	Distill-txtNER	text NER	E2E-NER	pLabel 4-gram
Pre-ASR	n/a	n/a	ftune 4-gram
Table 2: Methods for using external data for E2E models. Details are provided in Sec. 3.3.
So we may expect self-training from pseudo-labels to improve the target models by distilling the LM information into all model layers.
When the two models are different, we refer to it as knowledge distillation (Hinton et al., 2014), where the information is being distilled from the labeling model to the target model. This approach enables the target model to learn from the better-performing labeling model via pseudo-labels. Among the baseline models, the pipeline performs better than E2E approaches, presumably since the former uses strong pre-trained text representations. So, for instance, distilling from the pipeline (labeling model) into the E2E model (target model) is expected to boost the performance of the E2E model.
The LMs used for decoding in different approaches are mentioned in Tab. 2. All the ASR experiments use language models trained on the TED-LIUM 3 LM corpus (Hernandez et al., 2018) as in Shon et al. (2022). The language model used in baseline E2E NER experiments is trained on the 15hr fine-tune set (ftune 4-gram). The generated pseudo-labels also provide additional annotated data for LM training, which can be used in E2E models. These are referred to as plabel 4-gram (for "pseudo-label 4-gram").
Unlabeled speech: The unlabeled speech is used to improve the ASR module of the pipeline approach via self-training (SelfTrain-ASR).
For improving the E2E model, the improved pipeline can be used as the labeling model, followed by training the E2E model on the generated pseudo-labels (Distill-Pipeline). Alternatively, the unlabeled audio can be directly used to improve the E2E model via self-training (SelfTrain-E2E).
Unlabeled text: The text NER module in the pipeline approach is improved by self-training using the unlabeled text data (SelfTrain-txtNER). The E2E model uses the pseudo labels generated from
the text NER baseline module on the unlabeled text to update the LM used for decoding (Distill-textNER-lm).
Transcribed speech: The pipeline approach is improved by using the additional transcribed speech data to improve the ASR module (Pre-ASR). The E2E model uses this updated ASR as an initialization in a typical transfer learning setup. Alternatively, for paired speech text data, the pseudolabels generated from the text NER model can be paired with audio and used for training the E2E model, thus distilling information from a stronger text NER model into it (Distill-txtNER).
Text NER data: In addition to improving the pipeline and E2E models using the approaches mentioned above, we also look for any possible improvements in the text NER model by leveraging a larger external annotated text NER corpus. The DeBERTa base model is first fine-tuned on the larger external corpus, and then further fine-tuned on the in-domain labeled data. The first fine-tuning step is expected to help avoid shortcomings in performance due to the limited size of the in-domain labeled data.
This approach is limited by the availability of external datasets with the same annotation scheme as the in-domain corpus. We use the OntoNotes5.0 (Pradhan et al., 2013) corpus, whose labeling scheme inspired that of VoxPopuli (Shon et al., 2022). See Tab. 3 for more information on OntoNotes5.0.
VoxPopuli (Wang et al., 2021a) is a large multilingual speech corpus consisting of European Parliament event recordings with audio, transcripts, and timestamps from the official Parliament website. The English subset of the corpus has 540 hours of
Data split	# Utt	Duration (hours)	# entity phrases
fine-tune	5k	15	5820
dev	1.7k	5	1862
test	1.8k	5	2006
ext-100h	350k	101	N/A
ext-500h	177k	508
ext-NER (ontonotes-train)	66.6k	N/A	81.8k
spoken data with text transcripts. Shon et al. (2022) recently published NE annotations for a 15-hour subset of the train set and the complete standard dev set. Test set annotations are not public, but we obtain test set results by submitting model outputs following the SLUE site instructions. For our experiments with external in-domain data, we use uniformly sampled 100-hour and 500-hour subsets of the remainder of the VoxPopuli train set. The statistics for these splits are reported in Tab. 3. For more information on NE tags and label distribution, we direct the reader to the dataset and annotation papers (Pradhan et al., 2013; Shon et al., 2022).
4.2 Baseline models
We closely follow the setup for E2E and pipeline baselines in Shon et al. (2022).<sup>4</sup> We use wav2vec 2.0 base (Baevski et al., 2020) and DeBERTabase (He et al., 2021) as the unsupervised pretrained models, which have 95M and 139M parameters respectively. The publicly available wav2vec2.0 base model is pre-trained on 960 hours of the LibriSpeech audiobooks corpus (Panayotov et al., 2015).
For baselines that do not use pre-trained representations, we utilize the DeepSpeech2 (DS2) toolkit $^{5}$  (Amodei et al., 2016). DS2 first converts audio files into spectrogram features. The model processes the spectrogram features through two 2-D convolutional layers followed by five bidirectional 2048-dim LSTM layers and a softmax layer.
The softmax layer outputs the probabilities for a sequence of characters. The model has 26M parameters and is trained with SpecAugment data augmentation (Park et al., 2019) and a character-level CTC objective. Following Shon et al. (2022), we train on the finer label set (18 entity tags) and evaluate on the combined version (7 entity tags).
4.3 Utilizing external data
We use fairseq library (Ott et al., 2019) to fine-tune wav2vec 2.0 models for the E2E NER and ASR tasks. We fine-tune the model for  $80\mathrm{k}$  (160k) updates on 100 (500) hours of pseudo-labeled data. It takes 20 (40) hours (wall clock time) to fine-tune on 100 (500) hours of data using 8 TITAN RTX GPUs. We use HuggingFace's transformers toolkit (Wolf et al., 2020) for training the text NER model on pseudo-labels. Detailed config files will be provided in the public codebase. $^{6}$
5.1 Baseline models
Table 3: Data statistics. The "ext-" prefix denotes external datasets. The external data doesn't have named entity annotations, except for OntoNotes 5.0.
NER system	Pretrained model	F1
Speech	Text
Pipeline	X	DeBERTa-B	52.4
E2E	X	X	51.8
Pipeline	W2V2-B	DeBERTa-B	72.0
E2E	W2V2-B	X	68.1
Text NER	X	DeBERTa-B	86.0
Table 4: Dev set  $\%$  f-score performance of baseline models. All models here are trained on the 15hr finetune set. The pre-trained speech and text models are mentioned wherever used or applicable. The last three rows are from previously established baselines (Shon et al., 2022).
Results from all the baseline models are reported in Tab. 4. The models here are trained on the 15hr fine-tune set. We see that self-supervised pre-training gives a significant performance boost over no pre-training. The text NER model (which uses ground-truth transcripts) is far better than the pipeline method, which is better than the E2E model.
5.2 Leveraging external data
We report F1 scores on the dev set using different pipeline and E2E approaches in Tables 5 and 6 re
$^{6}$ https://github.com/asappresearch/spoken-ner
$^{3}$ https://asappresearch.github.io/slue-toolkit/how-to-submit.html  
 $^{4}$ https://github.com/asappresearch/slue-toolkit/blob/main/README.md  
 $^{5}$ https://github.com/SeanNaren/deepspeech.pytorch
Ext. data	Method	100h	500h
Un-Sp	SelfTrain-ASR	73.8	74.4
Un-Txt	SelfTrain-txtNER	72.3	70.8
Sp-Txt	Pre-ASR	75.6	77.7
Table 5: Dev set  $\%$  f-score performance of the pipeline models. Note the baseline Pipeline (72) and text NER (86.0) performances without using any additional data from Tab. 4.
Ext. data	Method	100h	500h
Un-Sp	SelfTrain-E2E	70.6	72.1
Distill-Pipeline	76.5	77.5
Un-Txt	Distill-txtNER-lm	71.0	71.7
Sp-Txt	Distill-txtNER	79.2	82.2
Pre-ASR	70.7	73.2