SlowGuess/NAACL_2022_OCR huggingface dataset:
Table 6: Dev set  $\%$  f-score performance of the E2E models. Note the baseline E2E (68.1) and text NER (86.0) performances without using any additional data from Tab. 4.
spectively. Fig. 1 presents key results when using each external data type for both E2E and pipeline models. The key findings are:
(i) Using external data reduces the gap between spoken NER baselines and text NER.
(ii) With access to either unlabeled speech or transcribed speech, E2E models outperform pipeline models, whereas, for the baselines, the opposite holds.
(iii) Using unlabeled text gives the smallest boost among the three types of external data, and the pipeline approach performs better in that setting.
A summary of test set results is presented in Appendix A.1. The results follow the same trend as on the dev set.
5.2.1 External text NER data
We try to improve the text NER model by using the OntoNotes5.0 NER corpus (Pradhan et al., 2013). Fine-tuning DeBERTa-base on OntoNotes5.0 produces an F1 of  $60\%$  on the VoxPopuli dev set. Finetuning it further on VoxPopuli gives F1  $86\%$  on the dev set. Since we do not see any boost over the existing vanilla approach  $(86\%, \text{see Tab. 4})$ , we retain the original text NER model using only in-domain data and do not perform further experiments using the OntoNotes-finetuned model.
The baseline results are not surprising: The limited labeled data is not enough for the baseline E2E approach, but the pipeline model can leverage a strong text representation model, which gives it an edge. Improvements to these models can be attributed to either (i) a better speech-to-text conversion or (ii) a better semantic understanding of the input content. Next, we use this distinction to understand the observed improvements from using external data.
6.1 Improved E2E results
When using external data with the E2E model, the best performing methods use either (a) external unlabeled speech (Distill-Pipeline) or (b) transcribed speech (Distill-textNER). The labeling models have a stronger semantic component than the E2E baseline in both of these scenarios because of their strong text NER module. The same cannot be said for the other competing approaches for these external data categories, SelfTrain-NER and Pre-ASR, which provide much lower improvements. SelfTrain-NER distills information from the LM into the model layers, but the n-gram LM is much less powerful than the transformer-based text NER module used in Distill-Pipeline. The Pre-ASR approach has no means to improve the semantic component in the updated model.
In the presence of unlabeled text data, the modification comes from a better 4-gram LM trained on pseudo-labels. Note that the baseline E2E model parameters do not change, unlike when using the other two types of external data. This can explain why this approach only has a small improvement over the baseline.
6.2 Improved pipeline results
The baseline pipeline model already takes advantage of the text NER module, which leaves little room for improvement in the semantic understanding component. Specifically, using unlabeled text data to improve the text NER module (SelfTrain.txtNER) gives a small boost of  $0.4\%$ . For comparison, note that the improvement from using unlabeled speech is  $2.5\%$  over baseline. So, the hope with pipeline models is for the external data to improve the speech-to-text conversion, which can then help reduce error propagation between the independent pipeline modules.
6.3 Amount of external data
Almost all experiments produce a larger improvement when using 500 hours of external data than 100 hours. Only SelfTrain-textNER has a reverse trend (see Tab. 5). The higher amount of external data naturally increases the fraction of noisy data that the target model is trained on, and that may lead to a poorer model. We hypothesize that methods for balancing between the effects of manually annotated and pseudo-labeled examples could help tackle this issue (Park et al., 2020). However, we leave an in-depth investigation of this phenomenon to future work.
6.4 Error analysis
For analysis, we choose the best-performing models within each category.
Figure 3: 100-WER (\%) and NE ACC (\%) values on the dev set for the best-performing models in each category with access to 100 hours of external data.
Figure 4: Recall and precision on the dev set for the best-performing models in each category with access to 100 hours of external data.
Fig. 3 presents the NE accuracy and word error rates (WER). We strip off the tag-specific special character tokens when evaluating WER for the E2E
NER models. Note that we report 100-WER so that higher is better in both plots. We observe that the ASR used in pipeline models typically performs better than the speech-to-text conversion of E2E models, even when the former has a poorer F1 (Fig. 1). This may lead us to hypothesize that the E2E model recognizes NE words better while doing worse for other words. However, this hypothesis is not supported by the NE-ACC results (Fig. 3).
Next, we look at the breakdown of F1 into precision and recall (Fig. 4). We see that pipeline models have worse precision, thus suggesting that these suffer from a higher false-positive rate than the E2E models. This explains why NE-ACC is not predictive of F1; the former can inform us about errors due to false negatives, but not false positives.
6.4.1 Error categories
For a more detailed understanding of model behavior, we categorize the NER errors into an exhaustive list of types (details in Appendix A.2). We focus on four major categories showing noteworthy differences between pipeline and E2E approaches. We provide this analysis for the baselines, Distill-Pipeline, and SelfTrain-ASR models using external unlabeled speech data. The trends and observations presented here are consistent with the other two external data types.
The major error categories, along with examples, are presented in Fig. 5. We observe that:
(i) False detections are 1.5 times more common in pipeline models than in E2E models, as expected based on the lower precision for the former. This happens even when the falsely detected text is not a speech-to-text conversion error.
(ii) Over-detections are 3.5 to 4 times more common in the pipeline models even when the entity phrase is decoded correctly.
(iii) Missed detections for the E2E Distill-Pipeline model are drastically reduced compared to the E2E baseline. Missed detections refer to cases where the entity phrases are correctly transcribed but are not labeled as named entities. The improvement here therefore suggests that Distill-Pipeline improves the understanding capability of the E2E model, in addition to its speech-to-text capability. Also, note that the pipeline model does not enjoy the same benefit from unlabeled speech since this only involves self-training (instead of knowledge distillation from a much richer model for E2E).
Overall, the pipeline models suffer disproportionately from false positives. This seems to stem
Error category	Example
(see the sample below)	Category occurrence
ASR	NER	Pipeline	E2E	Text NER
The entity phrase is correctly decoded, but ...	... the prediction includes additional surrounding terms	(QUANT, 'only thirty three') in predicted	Correct ASR, over detection (%)
5	4.8	1.0
0	None	Un-Sp
... the entity phrase is not detected as a named entity	(QUANT, 'sixty seven') in GT	Correct ASR, missed detection (%)
15	4.5	5.4
10	5.0	Un-Sp
The predicted entity phrase is in the GT transcript, but ...	... the predicted entity phrase does not correspond to any GT entity (after accounting for partial overlaps and misspellings)	(LAW, 'monetary policy') in Predicted	Correct ASR, false detection (%)
10	8.8	8.1
5	5.4	6.0
None	Un-Sp	Un-Sp
The predicted entity phrase is not in the GT transcript, and ...	(ORG, 'ssn') in predicted	Incorrect ASR, false detection (%)
5.0	5.9	4.3
2.5	2.8	2.2
None	Un-Sp	Un-Sp
Ground-truth	monetary policy i saw that according to the recent poll the majority of icelanders still oppose eu membership since sixty seven are against and only thirty three in favour of accession	[('NORP', 'icelanders'), ('PLACE', 'eu'), ('QUANT', 'sixty seven'), ('QUANT', 'thirty three')]
Predicted	monetary policy i saw that according to the recent poll the majority of iceland still oppose eu membership since sixty seven are against and only thirty three in favour of ssn.	[('PLACE', 'iceland'), ('PLACE', 'eu'), ('QUANT', 'only thirty three'), ('LAW', 'monetary'), ('ORG', 'ssn')']
Figure 5: NER error category distribution on the dev set. The category-specific error rates in the plots are normalized by the total number of ground-truth (GT) entities. The examples here are artificially created from the same ground-truth example for ease of presentation. Actual examples of these categories are presented in Appendix A.2.
from the text NER model, which has even higher over-detection and false detection rates than the pipeline baseline models (Fig. 5). The reasons behind this difference between E2E and pipeline models need further investigation.
We have explored various ways to use different external data types that improve both pipeline and E2E methods for spoken NER. The best-performing model when using external data is an E2E approach. This is one of the few results in the literature thus far showing better performance for E2E over pipeline methods that use state-of-the-art modules for spoken language understanding. We develop some insights into this difference; we notice that pipeline models are adversely affected by false positives and that leveraging external data improves the semantic understanding capability of the E2E models.
We hope that our work provides guiding principles for researchers working on SLU tasks in similar low-resource domains when some form of external data is abundant. This work also leaves some interesting research questions for future work. For example, we see minor improvements between 100h and 500h of external data (see Tab. 5 and 6),
which suggests the question: What is the smallest amount of external data needed to obtain significant improvements in NER performance? Additionally, one preliminary experiment with external, out-of-domain text NER data (OntoNotes 5.0) fails to improve the text NER performance, suggesting the challenges of dealing with out-of-domain datasets. The scenario where we have access to out-of-domain external data is common but challenging, and warrants further study. From the modeling perspective, better fine-tuning strategies for wav2vec2.0 in low supervision settings have been proposed for ASR (Pasad et al., 2021); it would be interesting to explore how these findings may transfer to an SLU task.
We thank Yoav Artzi for his feedback and inputs to strengthen our experimental setup and analysis. We also thank the anonymous reviewers for the detailed comments and suggestions.
Dario Amodei, Sundaram Ananthanarayanan, Rishita Anubhai, Jingliang Bai, Eric Battenberg, Carl Case, Jared Casper, Bryan Catanzaro, Jingdong Chen, Mike Chrzanowski, Adam Coates, Greg Diamos,
Erich Olsen, Jesse H. Engel, Linxi Fan, Christopher Fougner, Awni Y. Hannun, Billy Jun, Tony Han, Patrick LeGresley, Xiangang Li, Libby Lin, Sharan Narang, Andrew Y. Ng, Sherjil Ozair, Ryan Prenger, Sheng Qian, Jonathan Raiman, Sanjeev Satheesh, David Seetapun, Shubho Sengupta, Chong Wang, Yi Wang, Zhiqian Wang, Bo Xiao, Yan Xie, Dani Yogatama, Jun Zhan, and Zhenyao Zhu. 2016. Deep speech 2: End-to-end speech recognition in english and mandarin. In Proceedings of the 33nd International Conference on Machine Learning, ICML 2016, New York City, NY, USA, June 19-24, 2016, volume 48 of JMLR Workshop and Conference Proceedings, pages 173-182. JMLR.org.
Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. 2020. wav2vec 2.0: A framework for self-supervised learning of speech representations. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual.
Emanuele Bastianelli, Andrea Vanzo, Pawel Swietojanski, and Verena Rieser. 2020. SLURP: A spoken language understanding resource package. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7252-7262, Online. Association for Computational Linguistics.
Lasse Borgholt, Jakob Drachmann Havtorn, Mostafa Abdou, Joakim Edin, Lars Maaløe, Anders Søgaard, and Christian Igel. 2021. Do we still need automatic speech recognition for spoken language understanding? ArXiv preprint, abs/2111.14842.
Antoine Caubrière, Sophie Rosset, Yannick Esteve, Antoine Laurent, and Emmanuel Morin. 2020. Where are we in named entity recognition from speech? In Proceedings of the 12th Language Resources and Evaluation Conference, pages 4514-4520, Marseille, France. European Language Resources Association.
Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017. Reading Wikipedia to answer open-domain questions. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1870-1879, Vancouver, Canada. Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Sahar Ghannay, Antoine Caubrière, Yannick Esteve, Nathalie Camelin, Edwin Simonnet, Antoine Laurent, and Emmanuel Morin. 2018. End-to-end
named entity and semantic concept extraction from speech. In SLT.
Alex Graves, Santiago Fernández, Faustino J. Gomez, and Jürgen Schmidhuber. 2006. Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks. In Machine Learning, Proceedings of the Twenty-Third International Conference (ICML 2006), Pittsburgh, Pennsylvania, USA, June 25-29, 2006, volume 148 of ACM International Conference Proceeding Series, pages 369-376. ACM.
Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2021. Deberta: decoding-enhanced bert with disentangled attention. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net.
François Hernandez, Vincent Nguyen, Sahar Ghannay, Natalia Tomashenko, and Yannick Esteve. 2018. TED-LIUM 3: Twice as Much Data and Corpus Repartition for Experiments on Speaker Adaptation. In SPECOM.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2014. Distilling the knowledge in a neural network. In NIPS Deep Learning Workshop.
Mohamed Ameur Ben Jannet, Olivier Galibert, Martine Adda-Decker, and Sophie Rosset. 2015. How to evaluate ASR output for named entity recognition? In Interspeech.
Xueli Jia, Jianzong Wang, Zhiyong Zhang, Ning Cheng, and Jing Xiao. 2020. Large-scale transfer learning for low-resource spoken language understanding. In *Interspeech* 2020, 21st Annual Conference of the International Speech Communication Association, Virtual Event, Shanghai, China, 25-29 October 2020, pages 1555-1559. ISCA.
Ji-Hwan Kim and Philip C Woodland. 2000. A rule-based named entity recognition system for speech input. In ICSLP.
Jing Li, Aixin Sun, Jianglei Han, and Chenliang Li. 2020a. A survey on deep learning for named entity recognition. IEEE Transactions on Knowledge and Data Engineering.
Xiaoya Li, Xiaofei Sun, Yuxian Meng, Junjun Liang, Fei Wu, and Jiwei Li. 2020b. Dice loss for data-imbalanced NLP tasks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 465-476, Online. Association for Computational Linguistics.
Samuel Louvan and Bernardo Magnini. 2018. Exploring named entity recognition as an auxiliary task for slot filling in conversational language understanding. In Proceedings of the 2018 EMNLP Workshop SCAI: The 2nd International Workshop on Search-Oriented Conversational AI, pages 74-80, Brussels, Belgium. Association for Computational Linguistics.
Loren Lusosch, Mirco Ravanelli, Patrick Ignoto, Vikrant Singh Tomar, and Yoshua Bengio. 2019. Speech model pre-training for end-to-end spoken language understanding. In *Interspeech* 2019, 20th Annual Conference of the International Speech Communication Association, Graz, Austria, 15-19 September 2019, pages 814-818. ISCA.
David Nadeau and Satoshi Sekine. 2007. A survey of named entity recognition and classification. Linguisticae Investigationes.
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations), pages 48-53, Minneapolis, Minnesota. Association for Computational Linguistics.
Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. 2015. Librispeech: An ASR corpus based on public domain audio books. In 2015 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2015, South Brisbane, Queensland, Australia, April 19-24, 2015, pages 5206-5210. IEEE.
Carolina Parada, Mark Dredze, and Frederick Jelinek. 2011. OOV sensitive named-entity recognition in speech. In Interspeech.
Daniel S. Park, William Chan, Yu Zhang, Chung-Cheng Chiu, Barret Zoph, Ekin D. Cubuk, and Quoc V. Le. 2019. Specaugment: A simple data augmentation method for automatic speech recognition. In Interspeech 2019, 20th Annual Conference of the International Speech Communication Association, Graz, Austria, 15-19 September 2019, pages 2613-2617. ISCA.
Daniel S. Park, Yu Zhang, Ye Jia, Wei Han, Chung-Cheng Chiu, Bo Li, Yonghui Wu, and Quoc V. Le. 2020. Improved noisy student training for automatic speech recognition. In Interspeech 2020, 21st Annual Conference of the International Speech Communication Association, Virtual Event, Shanghai, China, 25-29 October 2020, pages 2817-2821. ISCA.
Sree Hari Krishnan Parthasarathi and Nikko Strom. 2019. Lessons from building acoustic models with a million hours of speech. In IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2019, Brighton, United Kingdom, May 12-17, 2019, pages 6670-6674. IEEE.
Ankita Pasad, Ju-Chieh Chou, and Karen Livescu. 2021. Layer-wise analysis of a self-supervised speech representation model. In ASRU.
Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Hwee Tou Ng, Anders Björkelund, Olga Uryupina,
Yuchen Zhang, and Zhi Zhong. 2013. Towards robust linguistic analysis using OntoNotes. In Proceedings of the Seventeenth Conference on Computational Natural Language Learning, pages 143-152, Sofia, Bulgaria. Association for Computational Linguistics.
Lev Ratinov and Dan Roth. 2009. Design challenges and misconceptions in named entity recognition. In Proceedings of the Thirteenth Conference on Computational Natural Language Learning (CoNLL-2009), pages 147-155, Boulder, Colorado. Association for Computational Linguistics.
Christian Raymond. 2013. Robust tree-structured named entities recognition from speech. In ICASSP.
Ellen Riloff. 1996. Automatically generating extraction patterns from untagged text. In AAAI.
Steffen Schneider, Alexei Baevski, Ronan Collobert, and Michael Auli. 2019. wav2vec: Unsupervised pre-training for speech recognition. In Interspeech 2019, 20th Annual Conference of the International Speech Communication Association, Graz, Austria, 15-19 September 2019, pages 3465-3469. ISCA.
Henry Scudder. 1965. Probability of error of some adaptive pattern-recognition machines. IEEE Transactions on Information Theory.
Suwon Shon, Ankita Pasad, Felix Wu, Pablo Brusco, Yoav Artzi, Karen Livescu, and Kyu J Han. 2022. SLUE: New benchmark tasks for spoken language understanding evaluation on natural speech. In ICASSP.
Katsuhito Sudoh, Hajime Tsukada, and Hideki Isozaki. 2006. Incorporating speech recognition confidence into discriminative named entity recognition of speech data. In Proceedings of the 21st International Conference on Computational Linguistics and 44th Annual Meeting of the Association for Computational Linguistics, pages 617-624, Sydney, Australia. Association for Computational Linguistics.
Changhan Wang, Morgane Riviere, Ann Lee, Anne Wu, Chaitanya Talnikar, Daniel Haziza, Mary Williamson, Juan Pino, and Emmanuel Dupoux. 2021a. VoxPopuli: A large-scale multilingual speech corpus for representation learning, semisupervised learning and interpretation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 993-1003, Online. Association for Computational Linguistics.
Xinyu Wang, Yong Jiang, Nguyen Bach, Tao Wang, Zhongqiang Huang, Fei Huang, and Kewei Tu. 2021b. Automated concatenation of embeddings for structured prediction. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1:
Long Papers), pages 2643-2660, Online. Association for Computational Linguistics.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumont, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. 2020. Huggingface's transformers: State-of-the-art natural language processing. In EMNLP.
Qiantong Xu, Alexei Baevski, Tatiana Likhomanenko, Paden Tomasello, Alexis Conneau, Ronan Collobert, Gabriel Synnaeve, and Michael Auli. 2021. Self-training and pre-training are complementary for speech recognition. In ICASSP.
Qiantong Xu, Tatiana Likhomanenko, Jacob Kahn, Awni Hannun, Gabriel Synnaeve, and Ronan Collobert. 2020. Iterative pseudo-labeling for speech recognition. In *Interspeech* 2020, 21st Annual Conference of the International Speech Communication Association, Virtual Event, Shanghai, China, 25-29 October 2020, pages 1006-1010. ISCA.
Hemant Yadav, Sreyan Ghosh, Yi Yu, and Rajiv Ratn Shah. 2020. End-to-end named entity recognition from english speech. In Interspeech 2020, 21st Annual Conference of the International Speech Communication Association, Virtual Event, Shanghai, China, 25-29 October 2020, pages 4268-4272. ISCA.
Vikas Yadav and Steven Bethard. 2018. A survey on recent advances in named entity recognition from deep learning models. In Proceedings of the 27th International Conference on Computational Linguistics, pages 2145-2158, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
Shu-Wen Yang, Po-Han Chi, Yung-Sung Chuang, Cheng-I Jeff Lai, Kushal Lakhotia, Yist Y. Lin, Andy T. Liu, Jiatong Shi, Xuankai Chang, Guanting Lin, Tzu-Hsien Huang, Wei-Cheng Tseng, Kotik Lee, Da-Rong Liu, Zili Huang, Shuyan Dong, Shang-Wen Li, Shinji Watanabe, Abdelrahman Mohamed, and Hung-yi Lee. 2021. SUPERB: speech processing universal performance benchmark. In Interspeech 2021, 22nd Annual Conference of the International Speech Communication Association, Brno, Czechia, 30 August - 3 September 2021, pages 1194-1198. ISCA.
David Yarowsky. 1995. Unsupervised word sense disambiguation rivaling supervised methods. In 33rd Annual Meeting of the Association for Computational Linguistics, pages 189-196, Cambridge, Massachusetts, USA. Association for Computational Linguistics.
A.1 Results on the test set
We obtain test set results for our best-performing models, by submitting model outputs following the SLUE instructions. These results are presented in Fig. 6. We observe similar trends as on the dev set (see Fig. 1).
Figure 6: Spoken NER test set results with 100 hours of external data of different types. The "Baseline" and "Text NER" numbers are from Shon et al. (2022).
We can see from the precision and recall scores in Fig. 7, that our analytical conclusions about the pipeline model performing poorly due to false positives are consistent across these two splits.
Figure 7: Recall and precision on the test set for the best performing models using 100 hours of external data.
A.2 Error categories