Venue: NAACL 2022

# On the Use of External Data for Spoken Named Entity Recognition

Ankita Pasad\({}^{*}\) 2, Felix Wu\({}^{1}\), Suwon Shon\({}^{1}\), Karen Livescu\({}^{2}\), Kyu J. Han\({}^{1}\)

\({}^{1}\)ASAPP  2Toyota Technological Institute at Chicago

Footnote 2: footnotemark:

###### Abstract

Spoken language understanding (SLU) tasks involve mapping from speech signals to semantic labels. Given the complexity of such tasks, good performance is expected to require large labeled datasets, which are difficult to collect for each new task and domain. However, recent advances in self-supervised speech representations have made it feasible to consider learning SLU models with limited labeled data. In this work, we focus on low-resource spoken named entity recognition (NER) and address the question: Beyond self-supervised pre-training, how can we use external speech and/or text data that are not annotated for the task? We consider self-training, knowledge distillation, and transfer learning for end-to-end (E2E) and pipeline (speech recognition followed by text NER) approaches. We find that several of these approaches improve performance in resource-constrained settings beyond the benefits from pre-trained representations. Compared to prior work, we find relative improvements in F1 of up to 16%. While the best baseline model is a pipeline approach, the best performance using external data is ultimately achieved by an E2E model. We provide detailed comparisons and analyses, developing insights on, for example, the effects of leveraging external data on (i) different categories of NER errors and (ii) the switch in performance trends between pipeline and E2E models.

+
Footnote †: July 10-15, 2022 ©2022 Association for Computational Linguistics

## 1 Introduction

Named entity recognition (NER) is a popular task in natural language processing. It involves detecting the named entities and their categories from a text sequence. NER can be used to extract information from unstructured data, which can also be used as features for other tasks like question answering Chen et al. (2017) and slot filling for task-oriented dialogues Louvan and Magnini (2018).

Thanks to pre-trained text representations such as BERT Devlin et al. (2019), text-based NER has recently improved greatly Wang et al. (2021b); Li et al. (2020). _Speaken_ NER, on the other hand, is not as well-studied. It has the added challenges of continuous-valued and longer input sequences and, at the same time, provides opportunities to take advantage of acoustic cues in the input. A recent study Shon et al. (2022) shows that there is still 10-20% absolute degradation in the F1 scores of spoken NER models compared to text-based NER using gold transcripts (see Figure 1) despite using large pre-trained speech representation models. Closing this gap remains a critical challenge.

In this work, we study the potential benefits of using a variety of external data types: (a) plain speech audio, (b) plain text, (c) speech with transcripts, and (d) text-based NER data. We benchmark our findings against recently published baselines for NER on the VoxPopuli dataset of European Parliament speech recordings Shon et al. (2022) and also introduce baselines of our own. We observe improvement from leveraging every type of external data. Our analysis also quantifies the pros and cons of the pipeline (speech recognition followed

Figure 1: Improvements in spoken NER with 100 hours of external data of different types. “Pipeline” refers to approaches consisting of speech recognition followed by a text NER model; “E2E” refers to approaches that directly map from speech to NER-tagged text. The “Baseline” and “Text NER” numbers are from previously established baselines Shon et al. (2022).



by text NER) and end-to-end (E2E) approaches. The key improvements are summarized in Figure 1. Specific contributions include:

(i) Unlike previous work, we devote equal effort to improving both pipeline and E2E approaches.

(ii) We present experiments using various external data types and modeling approaches.

(iii) Overall, we obtain F1 improvements of up to 16% for the E2E model and 6% for the pipeline model over previously published baselines, setting a new state of the art for NER on this dataset.

(iv) We benchmark the advantage of self-supervised representations (SSR) over a baseline that uses standard spectrogram features. SSR gives relative improvements of 36%/31% for pipeline/E2E models, respectively. To our knowledge, prior work has not directly measured this improvement over competitive baselines tuned for the task.

(v) We establish that E2E models outperform pipeline approaches on this task, given access to external data, while the baseline models without the external data have the opposite relationship.

(vi) We provide a detailed analysis of model behavior, including differences in error types between pipeline and E2E approaches and the reasoning for the superiority of E2E over pipeline models when using external data but not in the baseline setting.

## 2 Related work

### Spoken named entity recognition

Relatively little work has been conducted on spoken NER Kim and Woodland (2000); Sudoh et al. (2006); Parada et al. (2011); Ghannay et al. (2018); Caubriere et al. (2020); Yadav et al. (2020); Shon et al. (2022) as compared to the extensively studied task of NER on text Nadeau and Sekine (2007); Ratinov and Roth (2009); Yadav and Bethard (2018); Li et al. (2020). While spoken NER is commonly done through a pipeline approach Sudoh et al. (2006); Raymond (2013); Jannet et al. (2015), there is rising interest in E2E approaches in the speech community Ghannay et al. (2018); Caubriere et al. (2020); Yadav et al. (2020); Shon et al. (2022). These two approaches are depicted in Fig. 2.

An early E2E spoken NER model was introduced by Ghannay et al. (2018). The approach is based on the DeepSpeech2 Amodei et al. (2016) architecture, with the addition of special characters for NER labels around the named entities in the transcription, and is trained with character-level connectionist temporal classification (CTC) Graves et al. (2006). Yadav et al. (2020) introduced an English speech NER dataset and proposed an E2E approach similar to Ghannay et al. (2018). They show that LM fusion improves the performance of the E2E approach. Caubriere et al. (2020) provided a detailed comparison between E2E and pipeline models; however, they focused on small RNN/CNN models and did not use state-of-the-art SSR models. All these approaches use at least 100 hours of annotated data.

These previous efforts have shown that E2E models can outperform pipeline approaches in a fully supervised setting. Borgholt et al. (2021) also made the same observation on a simplified NER task. However, these studies do not account for improvements in NLP from self-supervised text representations for their pipeline counterparts. Shon et al. (2022) introduced and worked with a low-resource NER corpus and showed that E2E models still do not rival pipeline approaches when state-of-the-art pre-trained models are used.

When using pre-trained representations, E2E models are at a disadvantage since the pipeline model also has access to a text model trained on \(>\)50GB of text, in addition to the same speech representation model as E2E. This inspires us to study the benefits of using additional unlabeled data.

We choose to work with the NER-annotated VoxPopuli corpus Wang et al. (2021); Shon et al. (2022). VoxPopuli consists of naturally spoken speech, unlike Bastianelli et al. (2020), and is annotated manually, unlike Yadav et al. (2020) and Borgholt et al. (2021) who obtain ground-truth labels using text model predictions. The SLUE benchmark Shon et al. (2022) is aimed at low-resource SLU and includes annotations for only 15 hours of data; this matches with the goals of our work, making it an ideal choice to benchmark our findings.

### Self-supervised pre-training

There is a long history of using unsupervised pre-training in NLP to improve performance over limited-data supervised training on a broad range of tasks. SSRs have started to make an impact on speech tasks as well, with the first improvements seen in large-scale ASR with wav2vec Schneider et al. (2019). More recently, improvements have been seen on more tasks with wav2vec 2.0 Baevski et al. (2020) and other models Yang et al. (2021) 

However, it is not yet clear how universal these pre-trained representations are for speech tasks, particularly for semantic understanding tasks like NER. Some pre-trained models achieve impressive performance across a variety of tasks, including some understanding tasks (Yang et al., 2021) and analyses suggest that they contain at least some word meaning information (Pasad et al., 2021). However, these pre-trained models have not yet been tested on a broad range of challenging understanding tasks. We believe our work is the first to quantify the improvements from SSR, specifically on spoken NER.

### Leveraging external data

_Self-training_(Scudder, 1965; Yarowsky, 1995; Riloff, 1996) is a popular approach to improve supervised models when some additional unannotated data is available. Self-training has been observed to improve ASR (Parthasarathi and Strom, 2019; Xu et al., 2021) and is also complementary to pre-training (Xu et al., 2021). To the best of our knowledge, this is the first work to introduce it to spoken NER while also studying its effects on both E2E and pipeline approaches.

_Knowledge distillation_ is widely used in model compression research. In this approach, some intermediate output from a teacher model is used to train a smaller student model (Hinton et al., 2014). In the context of our work, the teacher and student networks are two different approaches to solving NER tasks, and the latter is trained on the final output tags of the former.

_Transfer learning_ has been widely employed for SLU tasks (Lugosch et al., 2019; Jia et al., 2020), including E2E spoken NER (Ghannay et al., 2018; Caubriere et al., 2020). Automatic speech recognition (ASR) is a typically chosen task for pre-training a model before fine-tuning it for SLU. This choice is facilitated by the wider availability of transcribed speech than SLU annotations. Specifically for NER, ASR pre-training is expected to help since the accuracy of decoded texts can directly affect the final NER predictions.

## 3 Methods

Spoken NER involves detecting the entity phrases in a spoken utterance along with their tags. The annotations include the text transcripts for the audio and the entity phrases with their corresponding tags. Spoken NER, like any other SLU task, is typically tackled using one of two types of approaches: (i) Pipeline and (ii) End-to-end (E2E). As shown in Fig. 2, a pipeline approach decodes speech to text using ASR and then passes the decoded text through a text NER module, whereas an E2E system directly maps the input speech to the output task labels. Each approach has its own set of advantages and shortcomings. Pipeline systems can enjoy the individual advances from both the speech and the text research communities, whereas combining two modules increases inference time, and propagation of ASR errors can have unexpected detrimental effects on the text NER module performance. On the other hand, E2E models directly optimize a task-specific objective and tend to have faster inference. However, such models typically require a large amount of task-specific labeled data to perform well. This can be seen from previous papers on E2E NER (Yadav et al., 2020; Ghannay et al., 2018), where using at least 100 hours of labeled data is typical.

Figure 2: High-level summary of approaches typically used to solve spoken and textual NER tasks. Optional LM decoding is applied in ASR and E2E-NER models.

 



### Baseline models

The baselines we use for E2E and pipeline models are taken from Shon et al. (2022). Similarly to previous work (Shon et al., 2022; Ghannay et al., 2018; Yadav et al., 2020), we formulate E2E NER as character-level prediction with tag-specific special characters delimiting entity phrases. For example, the phrases "irish" and "eu" are tagged as NORP1 (\(\$\)) and GPE2 (\(\%\)) respectively in "_the \(\$\) irish ] system works within a legal and regulatory policy directive framework dictated by the \(\%\) eu j_".

Footnote 1: NORP: Nationalities or religious or political groups

Footnote 2: GPE: Countries, cities, states

The E2E NER and ASR modules are initialized with the wav2vec2.0 base (Baevski et al., 2020) pre-trained speech representation, while the text NER module is pre-trained with DeBERTa base (He et al., 2021). These pre-trained models are then fine-tuned for ASR/NER after adding a linear layer on top of the final hidden-state output.

Since text transcripts are typically a part of the NER annotations, we can also train an NER model that uses the ground-truth text as input. This text NER model serves roughly as a polpline and is further used in experiments with external data. The E2E NER and ASR models are trained with a character-level CTC objective. The text NER model is trained for token-level classification with cross-entropy loss.

It is expected that using self-supervised representations gives a significant boost in limited labeled data settings. In order to quantify the benefits of the pre-trained representations in our setting, we also report the performance of E2E and pipeline baselines that are trained from scratch, not utilizing any pre-trained models.

### Evaluation metrics

Similarly to previous work (Ghannay et al., 2018; Yadav et al., 2020), we evaluate performance using _micro-averaged F1_ scores on an unordered list of tuples of named entity phrase and tag pairs predicted for each sentence. An entity prediction is considered correct if both the entity text and the entity tag are correct.

Spoken NER introduces an added variability to the possible model errors due to speech-to-text conversion. We report word error rate (WER) to evaluate this aspect. _WER_ is the word-level Levenshtein distance between the ground-truth text and the decoded text generated by the model. Additionally, to get an idea of the errors made by the model specifically on named entities, we also evaluate _NE ACC_, the proportion of entity phrases correctly decoded in the speech-to-text conversion. An entity phrase is considered accurate only if all the words in the phrase are correctly decoded in the right order.

### Utilizing external data

Next, we describe our approaches that use data external to the task-specific labeled data to improve both the pipeline and the E2E models for spoken NER. We consider four types of external data: (i) unlabeled speech (_Un-Sp_), (ii) unlabeled text (_Un-Txt_), (iii) transcribed speech (_Sp-Txt_), and (iv) text-based NER data.

The majority of techniques we consider involve labeling the external data with a _labeling model_ (typically one of the baseline models) to produce _pseudo-labels_. The _target model_ is then trained on these generated pseudo-labels along with the original labeled NER data. Tables 1 and 2 present a detailed list of all methods we consider for improving pipeline and E2E models respectively. The methods we include use the first three kinds of external data listed above. The fourth kind, external text-based NER data, is used in experiments attempting to improve the text NER model; since it does not succeed (Sec. 5.2.1), this data source is not explored further for the pipeline and E2E models.

When the labeling model is the same as the target model, this is a well-established process called self-training (Scudder, 1965; Yarowsky, 1995; Riloff, 1996; Xu et al., 2020, 2021). In our setting, a word-level language model (LM) is used for decoding both the ASR and E2E NER models. Shou et al. (2022) observed that a LM consistently improves performance of all of the baseline models.

\begin{table}
\begin{tabular}{l l l} \hline \hline
**External** & \multirow{2}{*}{**Method**} & **Target** \\
**data type** & & **model** \\ \hline Un-Sp & SelfTrain-ASR & ASR \\ Un-Txt & SelfTrain-txtNER & text NER \\ Sp-Txt & Pre-ASR & ASR \\ \hline \hline \end{tabular}
\end{table}
Table 1: Methods for using external data for pipeline models. For “SelfTrain” approaches, the _labeling model_ is the same as the _target model_. The method for external transcribed data (_Sp-Tx_) is based on transfer learning and thus there is no _labeling model_. More details are provided in Sec. 3.3.



So we may expect self-training from pseudo-labels to improve the target models by distilling the LM information into all model layers.

When the two models are different, we refer to it as knowledge distillation Hinton et al. (2014), where the information is being distilled from the labeling model to the target model. This approach enables the target model to learn from the better-performing labeling model via pseudo-labels. Among the baseline models, the pipeline performs better than E2E approaches, presumably since the former uses strong pre-trained text representations. So, for instance, distilling from the pipeline (labeling model) into the E2E model (target model) is expected to boost the performance of the E2E model.

The LMs used for decoding in different approaches are mentioned in Tab. 2. All the ASR experiments use language models trained on the TED-LIUM 3 LM corpus Hernandez et al. (2018) as in Shon et al. (2022). The language model used in baseline E2E NER experiments is trained on the 15hr fine-tune set (_future 4-gram_). The generated pseudo-labels also provide additional annotated data for LM training, which can be used in E2E models. These are referred to as _plabel 4-gram_) (for "pseudo-label 4-gram").

**Unlabeled speech:** The unlabeled speech is used to improve the ASR module of the pipeline approach via self-training (_SelfTrain-ASR_).

For improving the E2E model, the improved pipeline can be used as the labeling model, followed by training the E2E model on the generated pseudo-labels (_Distill-Pipeline_). Alternatively, the unlabeled audio can be directly used to improve the E2E model via self-training (_SelfTrain-E2E_).

**Unlabeled text:** The text NER module in the pipeline approach is improved by self-training using the unlabeled text data (_SelfTrain-txtNER_). The E2E model uses the pseudo labels generated from the text NER baseline module on the unlabeled text to update the LM used for decoding (_Distill-txtNER-lm_).

**Transcribed speech:** The pipeline approach is improved by using the additional transcribed speech data to improve the ASR module (_Pre-ASR_). The E2E model uses this updated ASR as an initialization in a typical transfer learning setup. Alternatively, for paired speech text data, the pseudo-labels generated from the text NER model can be paired with audio and used for training the E2E model, thus distilling information from a stronger text NER model into it (_Distill-txtNER_).

**Text NER data:** In addition to improving the pipeline and E2E models using the approaches mentioned above, we also look for any possible improvements in the text NER model by leveraging a larger external annotated text NER corpus. The DeBERTa base model is first fine-tuned on the larger external corpus, and then further fine-tuned on the in-domain labeled data. The first fine-tuning step is expected to help avoid shortcomings in performance due to the limited size of the in-domain labeled data.

This approach is limited by the availability of external datasets with the same annotation scheme as the in-domain corpus. We use the OntoNotes5.0 Pradhan et al. (2013) corpus, whose labeling scheme inspired that of VoxPopuli Shon et al. (2022). See Tab. 3 for more information on OntoNotes5.0.

## 4 Experimental setup

### Dataset

VoxPopuli Wang et al. (2021) is a large multilingual speech corpus consisting of European Parliament event recordings with audio, transcripts, and timestamps from the official Parliament website. The English subset of the corpus has 540 hours of

\begin{table}
\begin{tabular}{l l l l l} \hline \hline
**External data type** & **Method** & **Labeling model** & **Target model** & **LM for decoding** \\ \hline \multirow{2}{*}{Un-Sp} & SelfTrain-E2E & E2E-NER & E2E-NER & pLabel 4-gram \\  & Distill-Pipeline & Pipeline-NER & E2E-NER & pLabel 4-gram \\  & (after SelfTrain-ASR) & & & \\ \hline Un-Txt & Distill-txtNER-lm & text NER & n/a & pLabel 4-gram \\ \hline Sp-Txt & Distill-txtNER & text NER & E2E-NER & pLabel 4-gram \\  & Pre-ASR & n/a & n/a & fune 4-gram \\ \hline \hline \end{tabular}
\end{table}
Table 2: Methods for using external data for E2E models. Details are provided in Sec. 3.3.

 

spoken data with text transcripts. shon2022recently published NE annotations for a 15-hour subset of the train set and the complete standard dev set. Test set annotations are not public, but we obtain test set results by submitting model outputs following the SLUE site instructions.3 For our experiments with external in-domain data, we use uniformly sampled 100-hour and 500-hour subsets of the remainder of the VoxPopuli train set. The statistics for these splits are reported in Tab. 3. For more information on NE tags and label distribution, we direct the reader to the dataset and annotation papers Pradhan et al. (2013); Shon et al. (2022).

Footnote 3: [https://asappresearch.github.io/slue-toolkit/how-to-submit.html](https://asappresearch.github.io/slue-toolkit/how-to-submit.html)

### Baseline models

We closely follow the setup for E2E and pipeline baselines in shon2022recursive.4 We use wav2vec 2.0 base Baevski et al. (2020) and DeBERTabase He et al. (2021) as the unsupervised pretrained models, which have 95M and 139M parameters respectively. The publicly available wav2vec2.0 base model is pre-trained on 960 hours of the LibriSpeech audiobooks corpus Panayotov et al. (2015).

Footnote 4: [https://github.com/asappresearch/sluetoothkit/blob/main/README.md](https://github.com/asappresearch/sluetoothkit/blob/main/README.md)

For baselines that do not use pre-trained representations, we utilize the DeepSpeech2 (DS2) toolkit5Amodei et al. (2016). DS2 first converts audio files into spectrogram features. The model processes the spectrogram features through two 2-D convolutional layers followed by five bidirectional 2048-dim LSTM layers and a softmax layer. The softmax layer outputs the probabilities for a sequence of characters. The model has 26M parameters and is trained with SpecAugment data augmentation Park et al. (2019) and a character-level CTC objective. Following shon2022recursive, we train on the finer label set (18 entity tags) and evaluate on the combined version (7 entity tags).

Footnote 5: [https://github.com/SeanNaren/deepspeech.pytorch](https://github.com/SeanNaren/deepspeech.pytorch)

### Utilizing external data

We use fairseq library Ott et al. (2019) to finetune wav2vec 2.0 models for the E2E NER and ASR tasks. We fine-tune the model for 80k (160k) updates on 100 (500) hours of pseudo-labeled data. It takes 20 (40) hours (wall clock time) to fine-tune on 100 (500) hours of data using 8 TITAN RTX GPUs. We use HuggingFace's transformers toolkit Wolf et al. (2020) for training the text NER model on pseudo-labels. Detailed config files will be provided in the public codebase.6

Footnote 6: [https://github.com/asappresearch/spoken-ner](https://github.com/asappresearch/spoken-ner)

## 5 Results

### Baseline models

Results from all the baseline models are reported in Tab. 4. The models here are trained on the 15hr fine-tune set. We see that self-supervised pre-training gives a significant performance boost over no pre-training. The text NER model (which uses ground-truth transcripts) is far better than the pipeline method, which is better than the E2E model.

### Leveraging external data

We report F1 scores on the dev set using different pipeline and E2E approaches in Tables 5 and 6 re

\begin{table}
\begin{tabular}{l|c c|r} \hline \hline
**NER** & \multicolumn{2}{c|}{**Pretrained model**} & \\
**system** & \multicolumn{1}{c}{**Speech**} & \multicolumn{1}{c|}{**Text**} & \\ \hline Pipeline & ✗ & DeBERTa-B & 52.4 \\ E2E & ✗ & ✗ & 51.8 \\ \hline Pipeline & W2V2-B & DeBERTa-B & 72.0 \\ E2E & W2V2-B & ✗ & 68.1 \\ \hline Text NER & ✗ & DeBERTa-B & 86.0 \\ \hline \hline \end{tabular}
\end{table}
Table 4: Dev set % f-score performance of baseline models. All models here are trained on the 15hr fine-tune set. The pre-trained speech and text models are mentioned wherever used or applicable. The last three rows are from previously established baselines Shon et al. (2022).

\begin{table}
\begin{tabular}{l|r r r} \hline \hline
**Data split** & **\# utt** & \begin{tabular}{c} **Duration** \\ **(hours)** \\ \end{tabular} & 
\begin{tabular}{c} **\# entity** \\ **phrases** \\ \end{tabular} \\ \hline fine-tune & 5k & 15 & 5820 \\ dev & 1.7k & 5 & 1862 \\ test & 1.8k & 5 & 2006 \\ \hline ext-100h & 350k & 101 & \\ ext-500h & 177k & 508 & N/A \\ ext-NER & \multirow{2}{*}{66.6k} & \multirow{2}{*}{N/A} & \multirow{2}{*}{81.8k} \\ (ontontones-train) & & & \\ \hline \hline \end{tabular}
\end{table}
Table 3: Data statistics. The “ext-” prefix denotes external datasets. The external data doesn’t have named entity annotations, except for OntoNotes 5.0.

 

spectively. Fig. 1 presents key results when using each external data type for both E2E and pipeline models. The key findings are:

(i) Using external data reduces the gap between spoken NER baselines and text NER.

(ii) With access to either unlabeled speech or transcribed speech, E2E models outperform pipeline models, whereas, for the baselines, the opposite holds.

(iii) Using unlabeled text gives the smallest boost among the three types of external data, and the pipeline approach performs better in that setting.

A summary of test set results is presented in Appendix A.1. The results follow the same trend as on the dev set.

#### 5.2.1 External text NER data

We try to improve the text NER model by using the OntoNotes5.0 NER corpus Pradhan et al. (2013). Fine-tuning DeBERTa-base on OntoNotes5.0 produces an F1 of 60% on the VoxPopuli dev set. Fine-tuning it further on VoxPopuli gives F1 86% on the dev set. Since we do not see any boost over the existing vanilla approach (86%, see Tab. 4), we retain the original text NER model using only in-domain data and do not perform further experiments using the OntoNotes-finetuned model.

## 6 Discussion and analysis

The baseline results are not surprising: The limited labeled data is not enough for the baseline E2E approach, but the pipeline model can leverage a strong text representation model, which gives it an edge. Improvements to these models can be attributed to either (i) a better speech-to-text conversion or (ii) a better semantic understanding of the input content. Next, we use this distinction to understand the observed improvements from using external data.

### Improved E2E results

When using external data with the E2E model, the best performing methods use either (a) external unlabeled speech (_Distill-Pipeline_) or (b) transcribed speech (_Distill-txtNER_). The labeling models have a stronger semantic component than the E2E baseline in both of these scenarios because of their strong text NER module. The same cannot be said for the other competing approaches for these external data categories, _SelfTrain-NER_ and _Pre-ASR_, which provide much lower improvements. _SelfTrain-NER_ distills information from the LM into the model layers, but the n-gram LM is much less powerful than the transformer-based text NER module used in _Distill-Pieling_. The _Pre-ASR_ approach has no means to improve the semantic component in the updated model.

In the presence of unlabeled text data, the modification comes from a better 4-gram LM trained on pseudo-labels. Note that the baseline E2E model parameters do not change, unlike when using the other two types of external data. This can explain why this approach only has a small improvement over the baseline.

### Improved pipeline results

The baseline pipeline model already takes advantage of the text NER module, which leaves little room for improvement in the semantic understanding component. Specifically, using unlabeled text data to improve the text NER module (_SelfTrain-txtNER_) gives a small boost of 0.4%. For comparison, note that the improvement from using unlabeled speech is 2.5% over baseline. So, the hope with pipeline models is for the external data to improve the speech-to-text conversion, which can then help reduce error propagation between the independent pipeline modules.

\begin{table}
\begin{tabular}{l|c|r r} \hline \hline
**Ext. data** & **Method** & **100h** & **500h** \\ \hline \multirow{2}{*}{Un-Sp} & SelfTrain-E2E & 70.6 & 72.1 \\  & Distill-Pipeline & 76.5 & 77.5 \\ \hline Un-Txt & Distill-txtNER-lm & 71.0 & 71.7 \\ \hline Sp-Txt & Distill-txtNER & 79.2 & 82.2 \\  & Pre-ASR & 70.7 & 73.2 \\ \hline \hline \end{tabular}
\end{table}
Table 6: Dev set % f-score performance of the E2E models. Note the baseline E2E (68.1) and text NER (86.0) performances without using any additional data from Tab. 4.

\begin{table}
\begin{tabular}{l|l|r r} \hline \hline
**Ext. data** & **Method** & **100h** & **500h** \\ \hline Un-Sp & SelfTrain-ASR & 73.8 & 74.4 \\ Un-Txt & SelfTrain-txtNER & 72.3 & 70.8 \\ Sp-Txt & Pre-ASR & 75.6 & 77.7 \\ \hline \hline \end{tabular}
\end{table}
Table 5: Dev set % f-score performance of the pipeline models. Note the baseline Pipeline (72) and text NER (86.0) performances without using any additional data from Tab. 4.

 



### Amount of external data

Almost all experiments produce a larger improvement when using 500 hours of external data than 100 hours. Only _SelfTrain-txtNER_ has a reverse trend (see Tab. 5). The higher amount of external data naturally increases the fraction of noisy data that the target model is trained on, and that may lead to a poorer model. We hypothesize that methods for balancing between the effects of manually annotated and pseudo-labeled examples could help tackle this issue Park et al. (2020). However, we leave an in-depth investigation of this phenomenon to future work.

### Error analysis

For analysis, we choose the best-performing models within each category.

Fig. 3 presents the NE accuracy and word error rates (WER). We strip off the tag-specific special character tokens when evaluating WER for the E2E NER models. Note that we report \(100-\text{WER}\) so that higher is better in both plots. We observe that the ASR used in pipeline models typically performs better than the speech-to-text conversion of E2E models, even when the former has a poorer F1 (Fig. 1). This may lead us to hypothesize that the E2E model recognizes NE words better while doing worse for other words. However, this hypothesis is not supported by the _NE-ACC_ results (Fig. 3).

Next, we look at the breakdown of F1 into precision and recall (Fig. 4). We see that pipeline models have worse precision, thus suggesting that these suffer from a higher false-positive rate than the E2E models. This explains why _NE-ACC_ is not predictive of F1; the former can inform us about errors due to false negatives, but not false positives.

#### 6.4.1 Error categories

For a more detailed understanding of model behavior, we categorize the NER errors into an exhaustive list of types (details in Appendix A.2). We focus on four major categories showing noteworthy differences between pipeline and E2E approaches. We provide this analysis for the baselines, _Distillipeline_, and _SelfTrain-ASR_ models using external unlabeled speech data. The trends and observations presented here are consistent with the other two external data types.

The major error categories, along with examples, are presented in Fig. 5. We observe that:

(i) False detections are 1.5 times more common in pipeline models than in E2E models, as expected based on the lower precision for the former. This happens even when the falsely detected text is not a speech-to-text conversion error.

(ii) Over-detections are 3.5 to 4 times more common in the pipeline models even when the entity phrase is decoded correctly.

(iii) Missed detections for the E2E _Distill-P Pipeline_ model are drastically reduced compared to the E2E baseline. Missed detections refer to cases where the entity phrases are correctly transcribed but are not labeled as named entities. The improvement here therefore suggests that _Distill-Pipeline_ improves the understanding capability of the E2E model, in addition to its speech-to-text capability. Also, note that the pipeline model does not enjoy the same benefit from unlabeled speech since this only involves self-training (instead of knowledge distillation from a much richer model for E2E).

Overall, the pipeline models suffer disproportionately from false positives. This seems to stem

Figure 4: Recall and precision on the dev set for the best-performing models in each category with access to 100 hours of external data.

Figure 3: \(100-\text{WER}\) (%) and NE ACC (%) values on the dev set for the best-performing models in each category with access to 100 hours of external data.



from the text NER model, which has even higher over-detection and false detection rates than the pipeline baseline models (Fig. 5). The reasons behind this difference between E2E and pipeline models need further investigation.

## 7 Conclusion

We have explored various ways to use different external data types that improve both pipeline and E2E methods for spoken NER. The best-performing model when using external data is an E2E approach. This is one of the few results in the literature thus far showing better performance for E2E over pipeline methods that use state-of-the-art modules for spoken language understanding. We develop some insights into this difference; we notice that pipeline models are adversely affected by false positives and that leveraging external data improves the semantic understanding capability of the E2E models.

We hope that our work provides guiding principles for researchers working on SLU tasks in similar low-resource domains when some form of external data is abundant. This work also leaves some interesting research questions for future work. For example, we see minor improvements between 100h and 500h of external data (see Tab. 5 and 6), which suggests the question: What is the smallest amount of external data needed to obtain significant improvements in NER performance? Additionally, one preliminary experiment with external, out-of-domain text NER data (OntoNotes 5.0) fails to improve the text NER performance, suggesting the challenges of dealing with out-of-domain datasets. The scenario where we have access to out-of-domain external data is common but challenging, and warrants further study. From the modeling perspective, better fine-tuning strategies for wav2vec2.0 in low supervision settings have been proposed for ASR Pasad et al. (2021); it would be interesting to explore how these findings may transfer to an SLU task.

## Acknowledgements

We thank Yoav Artzi for his feedback and inputs to strengthen our experimental setup and analysis. We also thank the anonymous reviewers for the detailed comments and suggestions.

## References

* Amodei et al. [2018] Dario Amodei, Sundaram Ananthanarayanan, Rishita Anubhai, Jingliang Bai, Eric Battenberg, Carl Case, Jared Casper, Bryan Catanzaro, Jingdong Chen, Mike Chrzanowski, Adam Coates, Greg Diamos,

Figure 5: NER error category distribution on the dev set. The category-specific error rates in the plots are normalized by the total number of ground-truth (GT) entities. The examples here are artificially created from the same ground-truth example for ease of presentation. Actual examples of these categories are presented in Appendix A.2.

 

Erich Elsen, Jesse H. Engel, Linxi Fan, Christopher Fougner, Awni Y. Hannun, Billy Jun, Tony Han, Patrick LeGresley, Xiangang Li, Libby Lin, Sharan Narang, Andrew Y. Ng, Sherjil Ozair, Ryan Prenger, Sheng Qian, Jonathan Raiman, Sanjeev Satheesh, David Seetapun, Shubho Sengupta, Chong Wang, Yi Wang, Zhiqian Wang, Bo Xiao, Yan Xie, Dani Yogatama, Jun Zhan, and Zhenyao Zhu. 2016. Deep speech 2 : End-to-end speech recognition in english and mandarin. In _Proceedings of the 33nd International Conference on Machine Learning, ICML 2016, New York City, NY, USA, June 19-24, 2016_, volume 48 of _JMLR Workshop and Conference Proceedings_, pages 173-182. JMLR.org.
* Baevski et al. (2020) Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. 2020. wav2vec 2.0: A framework for self-supervised learning of speech representations. In _Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual_.
* Bastianelli et al. (2020) Emanuele Bastianelli, Andrea Vanzo, Pawel Swietojanski, and Verena Rieser. 2020. SLURP: A spoken language understanding resource package. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 7252-7262, Online. Association for Computational Linguistics.
* Borgholt et al. (2021) Lasse Borgholt, Jakob Drachmann Havtorn, Mostafa Abdou, Joakim Edin, Lars Maaloe, Anders Sogaard, and Christian Igel. 2021. Do we still need automatic speech recognition for spoken language understanding? _ArXiv preprint_, abs/2111.14842.
* Caubriere et al. (2020) Antoine Caubriere, Sophie Rosset, Yannick Esteve, Antoine Laurent, and Emmanuel Morin. 2020. Where are we in named entity recognition from speech? In _Proceedings of the 12th Language Resources and Evaluation Conference_, pages 4514-4520, Marseille, France. European Language Resources Association.
* Chen et al. (2017) Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017. Reading Wikipedia to answer open-domain questions. In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 1870-1879, Vancouver, Canada. Association for Computational Linguistics.
* Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_, pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
* Ghannay et al. (2018) Sahar Ghannay, Antoine Caubriere, Yannick Esteve, Nathalie Camelin, Edwin Simonnet, Antoine Laurent, and Emmanuel Morin. 2018. End-to-end named entity and semantic concept extraction from speech. In _SLT_.
* Graves et al. (2006) Alex Graves, Santiago Fernandez, Faustino J. Gomez, and Jurgen Schmidhuber. 2006. Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks. In _Machine Learning, Proceedings of the Twenty-Third International Conference (ICML 2006), Pittsburgh, Pennsylvania, USA, June 25-29, 2006_, volume 148 of _ACM International Conference Proceeding Series_, pages 369-376. ACM.
* He et al. (2021) Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2021. Deberta: decoding-enhancedbert with disentangled attention. In _9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021_. OpenReview.net.
* Hernandez et al. (2018) Francois Hernandez, Vincent Nguyen, Sahar Ghanney, Natalia Tomashenko, and Yannick Esteve. 2018. TED-LIUM 3: Twice as Much Data and Corpus Repartition for Experiments on Speaker Adaptation. In _SPECOM_.
* Hinton et al. (2014) Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2014. Distilling the knowledge in a neural network. In _NIPS Deep Learning Workshop_.
* Jananet et al. (2015) Mohamed Ameur Ben Jannet, Olivier Galibert, Martine Adda-Decker, and Sophie Rosset. 2015. How to evaluate ASR output for named entity recognition? In _Interspeech_.
* Jia et al. (2020) Xueli Jia, Jianzong Wang, Zhiyong Zhang, Ning Cheng, and Jing Xiao. 2020. Large-scale transfer learning for low-resource spoken language understanding. In _Interspeech 2020, 21st Annual Conference of the International Speech Communication Association, Virtual Event, Shanghai, China, 25-29 October 2020_, pages 1555-1559. ISCA.
* Kim and Woodland (2000) Ji-Hwan Kim and Philip C Woodland. 2000. A rule-based named entity recognition system for speech input. In _ICSLP_.
* Li et al. (2020a) Jing Li, Aixin Sun, Jianglei Han, and Chenliang Li. 2020a. A survey on deep learning for named entity recognition. _IEEE Transactions on Knowledge and Data Engineering_.
* Li et al. (2020b) Xiaoya Li, Xiaofei Sun, Yukian Meng, Junjun Liang, Fei Wu, and Jiwei Li. 2020b. Dice loss for data-imbalanced NLP tasks. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 465-476, Online. Association for Computational Linguistics.
* Louvan and Magnini (2018) Samuel Louvan and Bernardo Magnini. 2018. Exploring named entity recognition as an auxiliary task for slot filling in conversational language understanding. In _Proceedings of the 2018 EMNLP Workshop SCAI: The 2nd International Workshop on Search-Oriented Conversational AI_, pages 74-80, Brussels, Belgium. Association for Computational Linguistics 

Loren Lugosch, Mirco Ravanelli, Patrick Ignoto, Vikrant Singh Tomar, and Yoshua Bengio. 2019. Speech model pre-training for end-to-end spoken language understanding. In Interspeech 2019, 20th Annual Conference of the International Speech Communication Association, Graz, Austria, 15-19 September 2019, pp. 814-818. Cited by: SS3.
* D. Nadeau and S. Sekine (2007)A survey of named entity recognition and classification. Lingvisticae Investigationes. External Links: Link Cited by: SS3.
* M. Ott, S. Edunov, A. Baevski, A. Fan, S. Gross, N. Ng, D. Grangier, and M. Auli (2019) fairseq: a fast, extensible toolkit for sequence modeling. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations), Minneapolis, Minnesota, pp. 48-53. External Links: ISBN 978-3-642-4655-3, Document, Link Cited by: SS3.
* V. Panayotov, G. Chen, D. Povey, and S. Khudanpur (2015) LIBRI speech: an ASR corpus based on public domain audio books. In 2015 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2015, South Brisbane, Queensland, Australia, April 19-24, 2015, pp. 5206-5210. External Links: Link Cited by: SS3.
* C. Parada, M. Dredze, and F. Jelinek (2011) OOV sensitive named-entity recognition in speech. In Interspeech, Cited by: SS3.
* D. S. Park, W. Chan, Y. Zhang, C. Chiu, B. Zoph, E. D. Cubuk, and Q. V. Le (2019) Specaugment: a simple data augmentation method for automatic speech recognition. In Interspeech 2019, 20th Annual Conference of the International Speech Communication Association, Graz, Austria, 15-19 September 2019, pp. 2613-2617. External Links: Link Cited by: SS3.
* D. S. Park, Y. Zhang, Y. Jia, W. Han, C. Chiu, B. Li, Y. Wu, and Q. V. Le (2020) Improved noisy student training for automatic speech recognition. In Interspeech 2020, 21st Annual Conference of the International Speech Communication Association, Virtual Event, Shanghai, China, 25-29 October 2020, pp. 2817-2821. External Links: Link Cited by: SS3.
* S. Hari Krishnan Parthasarathi and N. Strom (2019) Lessons from building acoustic models with a million hours of speech. In IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2019, Brighton, United Kingdom, May 12-17, 2019, pp. 6670-6674. External Links: Link Cited by: SS3.
* A. Pasad, J. Chou, and K. Livescu (2021) Layer-wise analysis of a self-supervised speech representation model. In ASRU, Cited by: SS3.
* S. Pradhan, A. Moschitti, N. Xue, H. Tou Ng, A. Bjorkelund, O. Uryupina, Y. Zhang, and Z. Zhong (2013) Towards robust linguistic analysis using OntoCNotes. In Proceedings of the Seventeenth Conference on Computational Natural Language Learning, Sofia, Bulgaria, pp. 143-152. External Links: Link Cited by: SS3.
* L. Ratinov and D. Roth (2009) Design challenges and misconceptions in named entity recognition. In Proceedings of the Thirteenth Conference on Computational Natural Language Learning (CoNLL-2009), Boulder, Colorado, pp. 147-155. External Links: Link Cited by: SS3.
* C. Raymond (2013) Robust tree-structured named entities recognition from speech. In ICASSP, Cited by: SS3.
* Ellen Riloff (1996) Automatically generating extraction patterns from untagged text. In AAAI, Cited by: SS3.
* S. Schneider, A. Baevski, R. Collobert, and M. Auli (2019) wav2vec: unsupervised pre-training for speech recognition. In Interspeech 2019, 20th Annual Conference of the International Speech Communication Association, Graz, Austria, 15-19 September 2019, pp. 3465-3469. External Links: Link Cited by: SS3.
* H. Scudder (1965) Probability of error of some adaptive pattern-recognition machines. IEEE Transactions on Information Theory. External Links: ISSN 1536-469X, Link, Document Cited by: SS3.
* S. Shon, A. Pasad, F. Wu, P. Brusco, Y. Artzi, K. Livescu, and K. J. Han (2022) SLUE: new benchmark tasks for spoken language understanding evaluation on natural speech. In ICASSP, Cited by: SS3.
* K. Sudoh, H. Tsukada, and H. Isozaki (2006) Incorporating speech recognition confidence into discriminative named entity recognition of speech data. In Proceedings of the 21st International Conference on Computational Linguistics and 44th Annual Meeting of the Association for Computational Linguistics, Sydney, Australia, pp. 617-624. External Links: Link Cited by: SS3.
* C. Wang, M. Riviere, A. Lee, A. Wu, C. Talnikar, D. Haziza, M. Williamson, J. Pino, and E. Dupoux (2021a) VoxPopuli: a large-scale multilingual speech corpus for representation learning, semi-supervised learning and interpretation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), Online, pp. 993-1003. External Links: Link Cited by: SS3.
* X. Wang, Y. Jiang, N. Bach, T. Wang, Z. Huang, F. Huang, and K. Tu (2021b) Automated concatenation of embeddings for structured prediction. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: 

Long Papers), Online. Association for Computational Linguistics.
* Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, et al. 2020. Huggingface's transformers: State-of-the-art natural language processing. In _EMNLP_.
* Xu et al. (2021) Qiantong Xu, Alexei Baevski, Tatiana Likhmanenko, Paden Tomasello, Alexis Conneau, Ronan Collobert, Gabriel Synnaeve, and Michael Auli. 2021. Self-training and pre-training are complementary for speech recognition. In _ICASSP_.
* Xu et al. (2020) Qiantong Xu, Tatiana Likhomanenko, Jacob Kahn, Awni Hannun, Gabriel Synnaeve, and Ronan Collobert. 2020. Iterative pseudo-labeling for speech recognition. In _Interspeech 2020, 21st Annual Conference of the International Speech Communication Association, Virtual Event, Shanghai, China, 25-29 October 2020_, pages 1006-1010. ISCA.
* Yadav et al. (2020) Hemant Yadav, Sreyan Ghosh, Yi Yu, and Rajiv Ratn Shah. 2020. End-to-end named entity recognition from english speech. In _Interspeech 2020, 21st Annual Conference of the International Speech Communication Association, Virtual Event, Shanghai, China, 25-29 October 2020_, pages 4268-4272. ISCA.
* Yadav and Bethard (2018) Vikas Yadav and Steven Bethard. 2018. A survey on recent advances in named entity recognition from deep learning models. In _Proceedings of the 27th International Conference on Computational Linguistics_, pages 2145-2158, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
* 3 September 2021_, pages 1194-1198. ISCA.
* Yarowsky (1995) David Yarowsky. 1995. Unsupervised word sense disambiguation rivaling supervised methods. In _33rd Annual Meeting of the Association for Computational Linguistics_, pages 189-196, Cambridge, Massachusetts, USA. Association for Computational Linguistics.

## Appendix A Appendix

### Results on the test set

We obtain test set results for our best-performing models, by submitting model outputs following the SLUE instructions.7 These results are presented in Fig. 6. We observe similar trends as on the dev set (see Fig. 1).

Footnote 7: [https://asappresearch.github.io/slue-toolkit/how-to-submit.html](https://asappresearch.github.io/slue-toolkit/how-to-submit.html)

We can see from the precision and recall scores in Fig. 7, that our analytical conclusions about the pipeline model performing poorly due to false positives are consistent across these two splits.

### Error categories

Fig. 8 illustrates via a flowchart our process of assigning the tuples in ground-truth and predicted outputs into different error categories. Tab. 7 presents examples for the four categories discussed in Sec 6.4. These are examples from the dev set, using the _Distill-Pipeline_ E2E model trained on 100 hours of data.

Figure 6: Spoken NER test set results with 100 hours of external data of different types. The “Baseline” and “Text NER” numbers are from [22].

Figure 7: Recall and precision on the test set for the best performing models using 100 hours of external data.

 



\begin{table}
\begin{tabular}{c|c|c} \hline
**Error category** & \multicolumn{2}{c}{**Outputs from E2E model**} \\ \cline{2-3}  & **GT** & **Predicted** \\ \hline Correct ASR, over detection & and this means that you look and tell us honestly what does it mean if you start @ three years ] later & this means that you look and tell us honestly what does it mean if you start @ three years later ] [(’WHEN’, ’three years’)] & this means that you look and tell us honestly what does it mean if you start @ three years later ] \\  & [(‘WHEN’, ‘three years’)] & [(’WHEN’, ’three years later’)] \\ \hline Correct ASR, missed detection & the situation in the \% drc ] is indeed terrible and it has been this way for quite a while and i am deeply concerned about the handling of the current issue with regard to the a province & the situation in the drc is indeed terrible and it has been this way for for quite a while and i am deeply concerned about the handling of the current issue with regard to the a province & the situation in the drc is indeed terrible and it has been this way for for quite a while and i am deeply concerned about the handling of the current issue with regard to the a province \\  & [(‘PLACE’, ’drc’), (‘PLACE’, ‘kasai’)] & [] \\ \hline Correct ASR, false detection & and yet @ one month ] after we adopted our compromise the council did not put it on the agenda did not even present it used this time to talk with the member states and the presidencies & still @ one month ] after we voted a compromise the ‘ council ] did not put it on the agenda did not even present it use this time it i used this time to talk with the member states and the presidency & a compromise the ‘ council ] did not put it on the agenda did not even present i use this time it i used this time to talk with the member states and the presidences & not put it on the agenda did not even present i use this time to talk with the member states and the presidency \\  & [(’WHEN’, ‘one month’)] & [(’WHEN’, ’one month’), (’ORG’, ‘council’)] \\ \hline Incorrect ASR, false detection & it has nothing to do with religion but it has all to do with patriarchy & it has nothing to do with religion but it has all to do with \% turkey ] \\ \hline \end{tabular}
\end{table}
Table 7: Qualitative examples for different error categories from the output of the E2E model using 100 hours of unlabeled speech (_Distill-Pipeline_).



* [14] M. C. Dickson, A. K. Sridharan, and R. J. Schuster, _Theoretical Computer Science: An Introduction to the Theory of Computing_, volume 2098 of _Lecture Notes in Computer Science_, pages 3-56. Springer, Berlin, Heidelberg, 2007.
* [15] P. B. W. Anderson, _A practical guide for the development of a new computer system_, volume 100 of _Lecture Notes in Computer Science_, pages 3-10. Springer, Berlin, Heidelberg, 2007.
* [16] G. E. Hagener, _On the complexity of the algorithm_, volume 100 of _Lecture Notes in Computer Science_, pages 3-10. Springer, Berlin, Heidelberg, 2007.

[MISSING_PAGE_POST]