Venue: NAACL 2025

# Medical Spoken Named Entity Recognition

Khai Le-Duc\({}^{1,2}\), David Thulke\({}^{4,5}\), Hung-Phong Tran\({}^{3}\), Long Vo-Dang\({}^{7}\),

**Khai Nguyen Nguyen\({}^{8}\), Truong-Son Hy\({}^{6}\), Ralf Schluter\({}^{4,5}\) \({}^{1}\)University of Toronto, Canada \({}^{2}\)University Health Network, Canada \({}^{3}\)Hanoi University of Science and Technology, Vietnam \({}^{4}\)Machine Learning and Human Language Technology Group, RWTH Aachen University, Germany \({}^{5}\)AppTek GmbH, Germany \({}^{6}\)University of Alabama at Birmingham, United States \({}^{7}\)University of Cincinnati, United States \({}^{8}\)College of William and Mary, United States duckhai.le@mail.utoronto.ca thy@uab.edu, {thulke,schlueter}@hltpr.rwth-aachen.de

###### Abstract

Spoken Named Entity Recognition (NER) aims to extract named entities from speech and categorise them into types like person, location, organization, etc. In this work, we present _VietMed-NER_ - the first spoken NER dataset in the medical domain. To our knowledge, our Vietnamese real-world dataset is the largest spoken NER dataset in the world regarding the number of entity types, featuring 18 distinct types. Furthermore, we present baseline results using various state-of-the-art pre-trained models: encoder-only and sequence-to-sequence; and conduct quantitative and qualitative error analysis. We found that pre-trained multilingual models generally outperform monolingual models on reference text and ASR output and encoders outperform sequence-to-sequence models in NER tasks. By translating the transcripts, the dataset can also be utilised for text NER in the medical domain in other languages than Vietnamese. All code, data and models are publicly available1.

Footnote 1: [https://github.com/ledduckhai/MultiMed/tree/master/VietMed-NER](https://github.com/ledduckhai/MultiMed/tree/master/VietMed-NER)

## 1 Introduction

Named Entity Recognition (NER) targets extracting named entities (NE) from text and categorizing them into types like person, location, organization, etc. Initially studied in written language, recent attention has turned to study spoken NER Cohn et al. (2019); Shon et al. (2022), which aims to extract semantic information from speech. However, spoken NER has limited literature compared to NER on written text data Yadav et al. (2020).

Spoken NER is particularly challenging, firstly due to the impact of word segmentation on results. The medical vocabulary poses difficulties with numerous confused monosyllabic and polysyllubic words. For instance, the word "dtong" alone could denote "sugar" (chemical), "street" (location), or be part of a compound word like "dutongieu hoa" - "gastrointestinal" (anatomy). This confusion has also been reported in Chinese spoken NER by Chen et al. (2022). Further, data quality control and annotation consistency have been problematic, with some entities tagged in one sentence but not in others, and full NEs inconsistently tagged as multiple sub-NEs Huyen and Luong (2016); Nguyen et al. (2018, 2020); Truong et al. (2021). Finally, obtaining accurate medical NER from natural speech is challenging due to the lack of punctuation Ertopcu et al. (2017), speech disfluencies Kim and Woolland (2000), context, and the complexity of medical terms.

As for the medical domain, to the best of our knowledge, there is no dataset available for medical spoken NER. The only related work we found, Cohn et al. (2019), published a NER evaluation benchmark using an English general-domain conversational dataset, Switchboard Godfrey et al. (1992) and Fisher Cieri et al. (2004), for the task of audio de-identification specifically targeting Personal Health Identifiers.

To address this gap, we introduce _VietMed-NER_, a medical spoken NER dataset built on the real-world medical Automatic Speech Recognition (ASR) dataset _VietMed_Le-Duc (2024), featuring 18 medically-defined entity types. In the era of the advanced in-context learning capabilities of Large Language Models (LLMs) and human-level text-to-speech technologies, the dataset, with entity positional labels maintained during translation, is applicable not only to Vietnamese but also to other languages (see Appendix C). This enables various real-world applications, including: search engines Rud et al. (2011), content classification for news providers Kumaran and Allan (2004), medical ASR error correction Mani et al. (2020), audio

de-identification (Cohn et al., 2019) and content recommendation systems (Koperski et al., 2017).

Our contributions are as follows:

* the first publicly-available medical spoken NER dataset.
* We present baselines on several state-of-the-art pre-trained models
* We conduct quantitative and qualitative error analysis for medical spoken NER in Vietnamese

All code, data and models are published online1.

Footnote 1: [https://github.com/facebook/vist-dataset-for-challenge](https://github.com/facebook/vist-dataset-for-challenge)

## 2 Related Works

Traditionally, spoken NER has been done using a pipeline methodology, also known as cascaded approach, starting with an ASR stage, followed by NER applied to the generated transcriptions (Jannet et al., 2017; Benaicha et al., 2024). Another variant of the cascaded approach involves embedding specific entity expressions into the lexicon, thereby improving the language model's ability to accurately recognize these expressions (Hatmi et al., 2013).

Besides, end-to-end NER has recently garnered some attention within the research community. This approach seeks to optimize ASR and NER processes simultaneously, offering a potentially more efficient alternative to traditional pipeline methods by harnessing the ability of trainable acoustic features. However, its accuracy advantage over the cascaded approach remains a subject of debate, and the end-to-end training setup introduces additional complexity (Tomashenko et al., 2019; Yadav et al., 2020).

## 3 Data

### Data Collection

We chose the _VietMed_ dataset Le-Duc (2024), the world's largest publicly available medical ASR dataset, for annotating NEs.

The original dataset is in Vietnamese. We annotate the Vietnamese version with the methodology described in Section 3.2 and automatically translate the transcripts to English together with transferring the NE annotation.

### Annotation Process

The annotation of medical NEs from real-world speech is challenging because of the missing punctuation, special characters and capitalized words in ASR transcripts, disfluencies and required medical knowledge. Entirely manual annotation of NEs like in VLSP dataset Huyen and Luong (2016); Nguyen et al. (2018, 2020) and PhoNER_COVID19 Truong et al. (2021) requires a large number of working hours, not to mention the difficulties in quality control and inconsistency as we found in their corpora. These inconsistencies include: i) Some entities tagged in one sentence are not tagged in another sentence, and ii) Full NEs are inconsistently tagged as multiple sub-NEs. The best approach to tag nested NEs is the subject of ongoing debate Muis and Lu (2017); Li et al. (2021). For simplicity, higher consistency and to reduce the annotation effort, we only annotate the largest and outermost full entity span.

Moreover, using fine-tuned models for pretagging doesn't apply to specific medical entity types. Similarly, using prompt engineering with large language models like GPT-4 for pre-tagging did not achieve acceptable accuracy. Training a seed model with a gazetteer list requires initial training time, subsequent repetitive training schedules, and may prove unreliable due to its statistical reliance on a small amount of low-resource data Kozareva (2006).

To tackle these problems, we conduct a human-machine annotation approach, as described below:

1. Annotate and categorize a set of initial entities, then add them to a gazetteer list.
2. Sort entities by character length from highest to lowest, to distinguish between sub-NEs and full NEs, ensuring full NEs are mapped before sub-NEs. Time complexity = \(O(k\cdot\log(k))\) where \(k\) is the number of NEs. For example, "tooth pain" should be mapped before "pain".
3. Automatically map entities from the gazetteer list to the transcript. Time complexity = \(O(m\cdot n)\), where \(m\) is the number of NEs in gazeter list, \(n\) is the number of sentences. Pseudo code:

``` forNEingazetteer_list: forseninsentences: ifNEinsen:annotate(NE,sen) 

4. Annotators review each sentence to include correctly labeled NEs and ignore mislabeled NEs
5. Annotators add new NEs not in the gazetteer list during manual annotation. Steps 2 and 3 generate pre-tagged labels in the next sentences. Annotators repeat Steps 4 and 5 until the entire corpus is annotated.

We experience faster annotation by allowing annotators to foresee possible NEs in upcoming utterances based on previously annotated ones. Annotators can accept or reject these suggestions, saving time with correct suggestions and easily ignoring incorrect ones. Unlike training a seed model with a gazetteer, which requires initial training time and may be unreliable for low-to-mid resource languages, our method avoids these issues and eliminates the need to correct incorrect NEs.

### Data Quality Control

We created initial annotation guidelines (see Appendix A) and began annotating the corpus. Two developers, one with a medical background, independently annotated the corpus. Then, we held a discussion session to resolve conflicts, address complex cases, and refine the guidelines. Two other developers perform quality control using the guidelines and the annotated corpus. We consistently revisited each sentence in the entire corpus multiple times. This data quality control process is inspired by Tran et al. (2022).

### Data Splitting

Most NER datasets have a very small number of entities in their test sets compared to train and dev set Huyen and Luong (2016); Truong et al. (2021); Chen et al. (2022). However, we want to leverage the capabilities of large pre-trained models which are trained on vast amounts of unlabeled text data, resulting in good representations. Therefore, we focus on creating a large test set to obtain more statistically significant evaluation results and keep the training set relatively small in comparison.

### Data Statistics

Table 1 shows the statistics of our dataset. Our _VietMed-NER_ contains 18 entity types across 9000 sentences, split into train-dev-test as 8-2-6 hours. To the best of our knowledge, compared to all other public spoken NER datasets, ours has the largest number of entity types.

## 4 Experimental Setups

We employ the cascaded (two-stage) pipeline for spoken NER: A hybrid ASR model transcribes audio into text and then the transcribed text is fed into a text NER model.

### Evaluation Metrics

We employed the F1 score metric as it is commonly used for spoken NER Shon et al. (2022); Benaicha et al. (2024), which evaluates an unordered list of NE phrases and tag pairs predicted for each sentence. We used 3 toolkits for a more comprehensive comparison, as described in Appendix D.

\begin{table}
\begin{tabular}{l|l|r r|r|r|r|r|r|r}  & \multicolumn{1}{c|}{**Definition**} & \multicolumn{1}{c|}{**Train**} & \multicolumn{1}{c|}{**Dev**} & \multicolumn{1}{c|}{**Test**} & \multicolumn{1}{c}{**All**} \\ \cline{3-10}  & & Total & Uni. & Total & Uni. & Total & Uni. & Total & Uni. \\ \hline AGE & Age of a person & 447 & 43 & 108 & 25 & 611 & 83 & 1166 & 151 \\ \hline GENDER & Gender of a person & 202 & 30 & 46 & 15 & 451 & 33 & 699 & 78 \\ \hline JOB & Job of a person & 543 & 32 & 133 & 16 & 562 & 43 & 1238 & 91 \\ \hline LOCATION & Locations and places & 284 & 66 & 76 & 31 & 317 & 75 & 677 & 172 \\ \hline ORGANIZATION & Organizations & 19 & 14 & 2 & 2 & 58 & 23 & 79 & 39 \\ DISEASESYMTOM & Symptoms and diseases & 2699 & 518 & 683 & 209 & 1334 & 357 & 4716 & 1084 \\ \hline DRUGCHMICAL & Bio-chemical substances and drugs & 1054 & 255 & 263 & 104 & 684 & 136 & 2001 & 495 \\ \hline FOODDRINK & Food and beverage & 243 & 77 & 48 & 26 & 247 & 43 & 538 & 146 \\ \hline ORGAN & Anatomical features, e.g. organs, cells & 1827 & 252 & 444 & 122 & 1190 & 172 & 3461 & 546 \\ \hline PERSONALCARE & Personal care, e.g. hygiene routines, skin care & 353 & 114 & 82 & 38 & 95 & 10 & 530 & 162 \\ \hline DIAGNOSTICS & Diagnostic procedures, e.g. lab tests, imaging & 371 & 53 & 91 & 25 & 292 & 36 & 754 & 114 \\ \hline TREATMENT & Non-surgical treatment, e.g. rehab. injection & 726 & 69 & 174 & 25 & 230 & 17 & 1130 & 111 \\ \hline SURGRY & Surgical procedures, e.g. implants, neurosurgery & 197 & 29 & 55 & 13 & 270 & 37 & 522 & 79 \\ PREVENTIVEMED & Preventive medicine & 341 & 53 & 80 & 25 & 18 & 6 & 439 & 84 \\ \hline MEDDEVICETECHNIQUE & Medical devices, instruments, and techniques & 324 & 84 & 67 & 30 & 603 & 144 & 994 & 258 \\ \hline UNITCALIRBATOR & Medical calibration, e.g. number of doses, calories & 800 & 155 & 215 & 75 & 251 & 106 & 1266 & 336 \\ \hline TRANSFPORATION & Means of transportation & 5 & 2 & 3 & 3 & 27 & 10 & 35 & 15 \\ \hline DATETIME & Date and time & 674 & 155 & 159 & 65 & 657 & 133 & 1490 & 353 \\ \hline \#Entities in total & & 11109 & 2001 & 2729 & 849 & 7897 & 1464 & 21735 & 4314 \\ \hline \#Sentences & & 4620 & & 1150 & & 3500 & & 9270 \\ \end{tabular}
\end{table}
Table 1: Entity definition and its statistics in our dataset. ”Uni.” means the number of unique entities.

 



### ASR Models

We employed two baseline models fine-tuned for ASR on _VietMed_ published by Le-Duc (2024): an acoustic monolingual pre-trained w2v2-Viet and an acoustic multilingual pre-trained XLSR-53-Viet model. w2v2-Viet model was pre-trained from scratch on 1204h of Vietnamese data. For the XLSR-53-Viet model, continued pre-training on 1204h of Vietnamese starting with XLSR-53 Conneau et al. (2021) was performed. Both have the same number of parameters (118M) and were finetuned on the same training set. Their WERs on the test set are 29.0% and 28.8% respectively.

### NER Models

Table 2 shows the statistics of various pre-trained monolingual and multilingual models we consider to fine-tune on our dataset. To our knowledge, these are the best pre-trained models that achieved state-of-the-art results on various downstream tasks in the Vietnamese language, including NER.

**Monolingual encoder models**: PhoBERT\({}_{base}\), PhoBERT\({}_{large}\), PhoBERT\({}_{base}\)-v2 Nguyen and Tuan Nguyen (2020), ViDeBERta\({}_{base}\)Tran et al. (2023).

**Monolingual sequence-to-sequence (seq2seq) models**: BARTpho Tran et al. (2022), ViT5 Phan et al. (2022),

**Multilingual encoder models**: XML-R\({}_{base}\), XML-R\({}_{large}\)Conneau et al. (2020).

**Multilingual seq2seq models**: mBART-50 Tang et al. (2020).

#### 4.3.1 Seq2seq Training for NER Task

Following the approach proposed by Phan et al. (2021) and later adopted by ViT5 Phan et al. (2022), we formulated the sequence tagging task as a sequence-to-sequence task by training the models to generate tags of labels before and after an entity token. In cases where the models fail to follow the mentioned format for an entity token, we use an "exception" tag, which will be later ignored during metric calculation, as the label.

#### 4.3.2 Training Hyperparameters

We used HuggingFace Transformers Wolf et al. (2019) for fine-tuning pre-trained models for the NER task. Vietnamese input sentences can be represented in either syllable or word level as described by Truong et al. (2021). However, we only employed word-level settings to train NER models. All our NER experiments were done by using the default hyperparameters by HuggingFace.

The default hyperparameters are as follows: Learning rate of 2e-5, linear learning rate scheduler, training batch size of 64, 50 training epochs, weight decay of 0.01, AdamW optimizer Loshchilov and Hutter (2019), Beta1 of 0.9, Beta2 of 0.999, and epsilon of 1e-8.

## 5 Experimental Results

Table 3 and 4 show results of NER using various pre-trained models. We observe that there was a performance drop in all models when evaluated on ASR transcripts, as expected due to the noisy nature of ASR output.

**1. Pre-trained multilingual models outperformed monolingual models, if multilingual models overcome the capacity dilution**: The pre-trained monolingual model PhoBERT\({}_{base}\)-v2 outperformed other monolingual models, at 0.74 of F1 score on reference text, and 0.57 on ASR output. Despite having fewer parameters than

\begin{table}
\begin{tabular}{|l|r|c|} \hline
**Model** & **\#Params** & **\#Data** \\ \hline PhoBERT\({}_{base}\) & 135M & \multirow{2}{*}{20GB} \\ \cline{1-2} PhoBERT\({}_{large}\) & 370M & \\ \hline PhoBERT\({}_{base}\)-v2 & 135M & 140GB \\ \hline ViDeBERTa\({}_{base}\) & 86M & 298GB \\ \hline XLM-R\({}_{base}\) & 270M & 2.5TB \\ \hline XLM-R\({}_{large}\) & 550M & 2.5TB \\ \hline mBART-50 & 611M & 3.9TB \\ \hline ViT5\({}_{base}\) & 310M & 888GB \\ \hline BARTpho & 396M & 20GB \\ \hline \end{tabular}
\end{table}
Table 2: Statistics of state-of-the-art pre-trained language models which we used for NER task.

\begin{table}
\begin{tabular}{|l|c|c|c|} \hline
**NER Model** & **Prec.** & **Rec.** & **F1** \\ \hline BARTPho & 0.64 & 0.73 & 0.68 \\ mBBART-50 & 0.64 & 0.66 & 0.65 \\ PhoBERT\({}_{base}\) & 0.67 & 0.78 & 0.72 \\ PhoBERT\({}_{base}\)-v2 & 0.68 & 0.79 & 0.74 \\ PhoBERT\({}_{large}\) & 0.69 & 0.77 & 0.73 \\ ViDeBERTa\({}_{base}\) & 0.50 & 0.41 & 0.45 \\ ViT5\({}_{base}\) & 0.64 & 0.74 & 0.69 \\ XLM-R\({}_{base}\) & 0.64 & 0.73 & 0.69 \\ XLM-R\({}_{large}\) & 0.71 & 0.77 & 0.74 \\ \hline \end{tabular}
\end{table}
Table 3: NER results on reference text of test set. The metrics shown are Precision, Recall, and overall micro F1 score. Results by entity types are shown in Tables 5-24 in the Appendix.



PhoBERT_large, it performed similarly, likely due to more pre-training data. The pre-trained multilingual model XLM-R_large achieved the best performance with an F1 score of 0.74 on reference text and 0.58 on ASR output, while XLM-R_base performed worse than PhoBERT_base-v2. This gap is explained by the larger pre-training data (2.5TB multilingual data for XLM-R vs. 140GB monolingual data for PhoBERT_base-v2). Our results with PhoBERT_base-v2 and XLM-R_large confirmed that pre-trained multilingual representations improve performance on medical spoken NER tasks, similar to other language-specific downstream tasks by Conneau et al. (2020); Liu et al. (2020). However, multilingual models may face a _Transfer-dilution Trade-off_Conneau et al. (2020), where they lack the capacity to learn effective multilingual representations. In other words, for a fixed sized model, the per-language performance decreases as we increase the number of languages Gurgurov et al. (2024). To address this trade-off, multilingual models should possess sufficient capacity, necessitating an adequately large model size Chen and Chen (2024). This is evident in the performance comparison between PhoBERT_base-v2 and XLM-R_base, as seen in other language-specific downstream tasks by Conneau et al. (2020); Arivazhagan et al. (2019).

**2. Encoder-based models outperform seq2seq models**: The best seq2seq model, BARTpho, achieved F1 scores of 0.68 on reference text and 0.53 on ASR output. Encoders generally performed better than seq2seq models, possibly because seq2seq's generative nature is less suited for classification tasks like NER.

## 3 Multi-lingual pre-training of the acoustic model does not affect cascaded NER performance

As expected by the similar WERs for the acoustic pre-trained monolingual model w2v2-Viet and the multilingual model XLSR-53-Viet, all NER models show comparable F1 scores, precision, and recall. This indicates that in addition to overall WER the models do not differ significantly in the recognition accuracy of medical NEs. Non-cascaded models might have advantages in utilising the additional pre-training data for the downstream task.

## 6 Error Analysis

We performed an error analysis using the best-performing models.

### Quantitative

We provide a detailed error analysis for each entity type across best models, utilizing three evaluation toolkits. The results are summarized in Tables 5-24 and Tables 25-44, with corresponding visual representations in the scatter plots shown in Figure 1 and Figure 2.

The top NER models showed high accuracy in recognizing OCCUPATION and TRANSPORTATION entities in both reference text and ASR output. Despite TRANSPORTATION having only 35 total and 15 unique samples, the best models performed well, likely due to the semantic clarity and predictable spans of these entities.

In contrast, PREVENTIVEMED showed higher misrecognition rates, despite sufficient sample size mitigating class imbalance. This may stem from two factors. First, preventive medicine terms often overlap with general medical terminology, making it difficult for the model to distinguish them from DRUGCHEMICAL or TREATMENT concepts. For example, "vaccination" is frequently misclassified as a therapeutic intervention (TREATMENT) in sentences that describe its role in disease prevention (PREVENTIVEMED). Also, models frequently struggle to differentiate between "vaccination" and "vaccine" (DRUGCHEMICAL). Second, preventive medicine involves long-term health

\begin{table}
\begin{tabular}{l|c|c|c|c|} \hline
**NER** & **ASR** & **Prec.** & **Rec.** & **F1** \\ \hline ViDeBERTa\_base & XLSR-53-Viet & 0.45 & 0.34 & 0.39 \\ \cline{2-5}  & w2v2-Viet & 0.45 & 0.34 & 0.39 \\ \hline ViT5\_base & XLSR-53-Viet & 0.52 & 0.46 & 0.48 \\ \cline{2-5}  & w2v2-Viet & 0.53 & 0.46 & 0.49 \\ \hline mBART-50 & XLSR-53-Viet & 0.35 & 0.05 & 0.09 \\ \cline{2-5}  & w2v2-Viet & 0.35 & 0.05 & 0.09 \\ \hline BARTpho & XLSR-53-Viet & 0.56 & 0.50 & 0.53 \\ \cline{2-5}  & w2v2-Viet & 0.55 & 0.50 & 0.52 \\ \hline PhoBERT\_base\_v2 & XLSR-53-Viet & 0.57 & 0.57 & 0.57 \\ \cline{2-5}  & w2v2-Viet & 0.58 & 0.56 & 0.57 \\ \hline PhoBERT\_base & XLSR-53-Viet & 0.56 & 0.56 & 0.56 \\ \cline{2-5}  & w2v2-Viet & 0.56 & 0.56 & 0.56 \\ \hline PhoBERT\_large & XLSR-53-Viet & 0.57 & 0.55 & 0.56 \\ \cline{2-5}  & w2v2-Viet & 0.58 & 0.55 & 0.56 \\ \hline XLM-R\_base & XLSR-53-Viet & 0.54 & 0.52 & 0.53 \\ \cline{2-5}  & w2v2-Viet & 0.54 & 0.52 & 0.53 \\ \hline XLM-R\_large & XLSR-53-Viet & 0.60 & 0.56 & 0.58 \\ \cline{2-5}  & w2v2-Viet & 0.60 & 0.56 & 0.58 \\ \hline \end{tabular}
\end{table}
Table 4: NER results on ASR output of test set for different NER and ASR models. Metrics shown are Precision, Recall, and overall micro F1 score. Results by entity types are shown in Tables 25-44 in the Appendix.

 

strategies often expressed in non-clinical or non-standardized language, complicating entity recognition.

### Qualitative

A common error was confusion between LOCATION and ORGANIZATION, due to the inherent ambiguity where the same entity can function as either depending on context. An organization-related entity may be labelled as LOCATION if it implies a patient visited there, but this inference requires external knowledge about the entity. Another confusion involved DRUGCHEMICAL and FOODDRINK. Both categories share similar names, descriptors, and consumption contexts (e.g., caffeine, alcohol, sugar). Insufficient context length often causes errors, especially with ambiguous terms like "vitamin," "cordyceps," or "sea daffodils," which can refer to both supplements and nutrients depending on context. Another case is DIAGNOSTICS, TREATMENT, and SURGERY. For example, a "biopsy" can be both a diagnostic and treatment, while "radiation therapy" may be linked to surgery.

A common error in NER involves incorrect entity spans, which fall into two types: (1) correct label but wrong span, and (2) wrong label but correct span. The first type often occurs with multi-word entities in the medical domain, like ORGANIZATION, LOCATION, or DISESASemyTOM. For example, "high blood pressure" (B-DISEasesSYMPTOM, I-DISEasesESYMPtOM) may be misrecognized as "high blood" (B-DISEasesSYMPTOM, I-DISEasesSYMPTOM, O), keeping the meaning but shortening the span. The second type occurs when compound-word entities are split, such as "vagina cells" (B-ORGAN, I-ORGAN) being misrecognized as "vagina" and "cells" (B-ORGAN, B-ORGAN).

Figure 1: Scatter plot of NER results on reference text by entity types using various pre-trained language models and evaluation variants, created by Tables 5-24.

Figure 2: Scatter plot of NER results on ASR output by entity types using various pre-trained language models and evaluation variants, created by Tables 25–44.

 

Conclusion

In this work, we present _VietMed-NER_ - the first spoken NER dataset in the medical domain. Our dataset contains 18 entity types, including both conventional and newly defined entity types for real-world medical conversations. Our results show that pre-trained multilingual models typically outperform monolingual models on both reference text and ASR output if the multilingual models are sufficiently large to learn multilingual representations. Additionally, encoders generally demonstrate better performance than seq2seq models in the NER task. Finally, while pre-trained audio data impacts ASR output, it does not significantly impact NER performance in the cascaded setting.

## 8 Limitations

**Our annotation approach**: Our annotation approach has some advantages over the fully manual approach. First, it allows annotators to not spend extra time tagging the entities that have been tagged in previous sentences. Second, it prevents that annotators miss entities that have been tagged in previous sentences, improving the consistency of the entire dataset. During our work, we experienced a faster annotation by using our approach compared to fully manual annotation. However, in the scope of this paper, we have not done extensive experiments to give a quantitative number of how much time has been saved and the method's impact on annotation quality.

**Evaluation metrics for medical terms**: ASR system performance is commonly evaluated using WER, which quantifies the ratio of word insertion, substitution, and deletion errors in a transcript relative to the total number of spoken words. However, various spoken language understanding tasks, such as spoken NER, rely on accurately identifying key terms within transcripts. In medical ASR, it is critical to account for the disproportionate importance of medical terms in doctor-patient interactions, as they hold significantly more weight than general vocabulary, as discussed in Section B in the Appendix. We believe that other domain-specific spoken NER tasks follow a similar pattern. Consequently, future comprehensive investigations into evaluation metrics are needed to determine the most appropriate metric for spoken NER in the medical and other domains.

## References

* Arivazhagan et al. (2019) Naveen Arivazhagan, Ankur Bapna, Orhan Firat, Dmitry Lepikhin, Melvin Johnson, Maxim Krikun, Mia Xu Chen, Yuan Cao, George Foster, Colin Cherry, et al. 2019. Massively multilingual neural machine translation in the wild: Findings and challenges. _ArXiv preprint_, abs/1907.05019.
* Benaicha et al. (2024) Moncef Benaicha, David Thulke, and Mehmet Ali Tugtekin Turan. 2024. Leveraging cross-lingual transfer learning in spoken named entity recognition systems. In _Proceedings of the 20th Conference on Natural Language Processing (KONVENS 2024)_, pages 98-105, Vienna, Austria. Association for Computational Linguistics.
* Chen et al. (2022) Boli Chen, Guangwei Xu, Xiaobin Wang, Pengjun Xie, Meishan Zhang, and Fei Huang. 2022. AISHELL-NER: named entity recognition from chinese speech. In _IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2022, Virtual and Singapore, 23-27 May 2022_, pages 8352-8356. IEEE.
* Chen et al. (2023) Li-Wei Chen, Shinji Watanabe, and Alexander Rudnicky. 2023. A vector quantized approach for text to speech synthesis on real-world spontaneous speech. In _Thirty-Seventh AAAI Conference on Artificial Intelligence, AAAI 2023, Thirty-Fifth Conference on Innovative Applications of Artificial Intelligence, IAAI 2023, Thirteenth Symposium on Educational Advances in Artificial Intelligence, EAAI 2023, Washington, DC, USA, February 7-14, 2023_, pages 12644-12652. AAAI Press.
* Chen and Chen (2024) Po-Heng Chen and Yun-Nung Chen. 2024. Efficient unseen language adaptation for multilingual pre-trained language models. In _Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing_, pages 18983-18994, Miami, Florida, USA. Association for Computational Linguistics.
* Cieri et al. (2004) Christopher Cieri, David Miller, and Kevin Walker. 2004. The fisher corpus: a resource for the next generations of speech-to-text. In _Proceedings of the Fourth International Conference on Language Resources and Evaluation (LREC'04)_, Lisbon, Portugal. European Language Resources Association (ELRA).
* a new entity recognition task. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Industry Papers)_, pages 197-204, Minneapolis, Minnesota. Association for Computational Linguistics.
* Conneau et al. (2021) Alexis Conneau, Alexei Baevski, Ronan Collobert, Abdelrahman Mohamed, and Michael Auli. 2021. Unsupervised cross-lingual representation learning for 

speech recognition. In _Interspeech 2021, 22nd Annual Conference of the International Speech Communication Association, Brno, Czechia, 30 August - 3 September 2021_, pages 2426-2430. ISCA. Conneau et al. (2020) Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzman, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 8440-8451, Online. Association for Computational Linguistics. Duan et al. (2017) Shaoyang Duan, Ruifang He, and Wenli Zhao. 2017. Exploiting document level information to improve event detection via recurrent neural networks. In _Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, pages 352-361, Taipei, Taiwan. Asian Federation of Natural Language Processing.
* Etropcu et al. (2017) Burak Etropcu, Ali Bugra Kanburoglu, Ozan Topsakal, Onur Acikoz, Ali Tunca Gurkan, Berke Ozenc, Ilker Cam, Begum Avar, Gokhan Ercan, and Olcay Taner Yildiz. 2017. A new approach for named entity recognition. In _2017 International Conference on Computer Science and Engineering (UBMK)_, pages 474-479. IEEE.
* Godfrey et al. (1992) John J Godfrey, Edward C Holliman, and Jane McDaniel. 1992. Switchboard: Telephone speech corpus for research and development. In _Proceedings of the 1992 IEEE International Conference on Acoustics, Speech and Signal Processing_.
* Gurgurov et al. (2024) Daniil Gurgurov, Tanja Baumel, and Tatiana Anikina. 2024. Multilingual large language models and curse of multilinguality. _ArXiv preprint_, abs/2406.10602.
* Hatmi et al. (2013) Mohamed Hatmi, Christine Jacquin, Emmanuel Morin, and Sylvain Meigner. 2013. Incorporating named entity recognition into the speech transcription process. In _Proceedings of the 14th Annual Conference of the International Speech Communication Association (Interspeech'13)_, pages 3732-3736.
* Huyen and Luong (2016) Nguyen Thi Minh Huyen and Vu Xuan Luong. 2016. Vlsp 2016 shared task: Named entity recognition. _Proceedings of Vietnamese Speech and Language Processing (VLSP)_.
* Jannet et al. (2017) Mohamed Areur Ben Jannet, Olivier Galibert, Martine Adda-Decker, and Sophie Rosset. 2017. Investigating the effect of ASR tuning on named entity recognition. In _Interspeech 2017, 18th Annual Conference of the International Speech Communication Association, Stockholm, Sweden, August 20-24, 2017_, pages 2486-2490. ISCA.
* Kim and Woodland (2000) Ji-Hwan Kim and Philip C Woodland. 2000. A rule-based named entity recognition system for speech input. In _Sixth International Conference on Spoken Language Processing_.
* Koperski et al. (2017) Krzysztof Koperski, Jisheng Liang, and Neil Roseman. 2017. Content recommendation based on collections of entities. US Patent 9,710,556.
* Kozareva (2006) Zornitsa Kozareva. 2006. Bootstrapping named entity recognition with automatically generated gazetteet lists. In _Student Research Workshop_, pages 15-22.
* Kumaran and Allan (2004) Giridhar Kumaran and James Allan. 2004. Text classification and named entities for new event detection. In _Proceedings of the 27th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval_, SIGIR '04, page 297-304, New York, NY, USA. Association for Computing Machinery.
* Le-Duc (2024) Khai Le-Duc. 2024. VietMed: A dataset and benchmark for automatic speech recognition of Vietnamese in the medical domain. In _Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LRECOLING 2024)_, pages 17365-17370, Torino, Italia. ELRA and ICCL.
* Levenshtein et al. (1966) Vladimir I Levenshtein et al. 1966. Binary codes capable of correcting deletions, insertions, and reversals. In _Soviet physics doklady_, volume 10, pages 707-710. Soviet Union.
* Li et al. (2021a) Fei Li, ZhiChao Lin, Meishan Zhang, and Donghong Ji. 2021a. A span-based model for joint overlapped and discontinuous named entity recognition. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, pages 4814-4828, Online. Association for Computational Linguistics.
* Li et al. (2021b) Fei Li, ZhiChao Lin, Meishan Zhang, and Donghong Ji. 2021b. A span-based model for joint overlapped and discontinuous named entity recognition. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, pages 4814-4828, Online. Association for Computational Linguistics.
* 16, 2023_.
* Liu et al. (2020) Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghavrinnejad, Mike Lewis, and Luke Zettlemoyer. 2020. Multilingual denoising pretraining for neural machine translation. _Transactions of the Association for Computational Linguistics_, 8:726-742 

Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In _7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019_. OpenReview.net.
* Mani et al. (2020) Anirudh Mani, Shruti Palaskar, and Sandeep Konam. 2020. Towards understanding ASR error correction for medical conversations. In _Proceedings of the First Workshop on Natural Language Processing for Medical Conversations_, pages 7-11, Online. Association for Computational Linguistics.
* Muis and Lu (2017) Aldrian Obaja Muis and Wei Lu. 2017. Labeling gaps between words: Recognizing overlapping mentions with mention separators. In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_, pages 2608-2618, Copenhagen, Denmark. Association for Computational Linguistics.
* Nguyen and Nguyen (2020) Dat Quoc Nguyen and Anh Tuan Nguyen. 2020. PhoBERT: Pre-trained language models for Vietnamese. In _Findings of the Association for Computational Linguistics: EMNLP 2020_, pages 1037-1042, Online. Association for Computational Linguistics.
* Nguyen et al. (2018) Huyen TM Nguyen, Quyen T Ngo, Luong X Vu, Yu M Tran, and Hien TT Nguyen. 2018. Vlsp shared task: Named entity recognition. _Journal of Computer Science and Cybernetics_.
* Nguyen et al. (2020) Thai Binh Nguyen, Quang Minh Nguyen, Hien Nguyen Thi Thu, Quoc Truong Do, and Luong Chi Mai. 2020. Improving vietnamese named entity recognition from speech using word capitalization and punctuation recovery models. In _Interspeech 2020, 21st Annual Conference of the International Speech Communication Association, Virtual Event, Shanghai, China, 25-29 October 2020_, pages 4263-4267. ISCA.
* Phan et al. (2022) Long Phan, Hieu Tran, Hieu Nguyen, and Trieu H. Trinh. 2022. ViT5: Pretrained text-to-text transformer for Vietnamese language generation. In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: Student Research Workshop_, pages 136-142, Hybrid: Seattle, Washington + Online. Association for Computational Linguistics.
* Phan et al. (2021) Long N. Phan, James T. Anibal, Hieu Tran, Shaurya Chanana, Erol Bahadroglu, Alec Peltekian, and Gregoire Altan-Bonnet. 2021. Sciffive: a text-to-text transformer model for biomedical literature.
* Rud et al. (2011) Stefan Rud, Massimiliano Ciaramita, Jens Muller, and Hinrich Schutze. 2011. Piggyback: Using search engines for robust cross-domain named entity recognition. In _Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies_, pages 965-975, Portland, Oregon, USA. Association for Computational Linguistics.
* Shon et al. (2023) Suwon Shon, Siddhatn Arora, Chyi-Jitunn Lin, Ankita Pasad, Felix Wu, Roshan S Sharma, Wei-Lun Wu, Hung-yj Lee, Karen Livescu, and Shinji Watanabe. 2023. SLUE phase-2: A benchmark suite of diverse spoken language understanding tasks. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 8906-8937, Toronto, Canada. Association for Computational Linguistics.
* Shen et al. (2022) Suwon Shon, Ankita Pasad, Felix Wu, Pablo Brusco, Yoav Artzi, Karen Livescu, and Kyu Jeong Han. 2022. SLUE: new benchmark tasks for spoken language understanding evaluation on natural speech. In _IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2022, Virtual and Singapore, 23-27 May 2022_, pages 7927-7931. IEEE.
* Strakova et al. (2019) Jana Strakova, Milan Straka, and Jan Hajic. 2019. Neural architectures for nested NER through linearization. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 5326-5331, Florence, Italy. Association for Computational Linguistics.
* Tan et al. (2024) Xu Tan, Jiawei Chen, Haohe Liu, Jian Cong, Chen Zhang, Yanqing Liu, Xi Wang, Yichong Leng, Yuanhao Yi, Lei He, et al. 2024. Nutarspeech: End-to-end text-to-speech synthesis with human-level quality. _IEEE Transactions on Pattern Analysis and Machine Intelligence_.
* Tang et al. (2020) Yuqing Tang, Chau Tran, Xian Li, Peng-Jen Chen, Nam Goyal, Vishrav Chaudhary, Jiatao Gu, and Angela Fan. 2020. Multilingual translation with extensible multilingual pretraining and finentuning.
* Tomashenko et al. (2019) Natalia Tomashenko, Antoine Caubriere, Yannick Essteve, Antoine Laurent, and Emmanuel Morin. 2019. Recent advances in end-to-end spoken language understanding. In _Statistical Language and Speech Processing: 7th International Conference, SLSP 2019, Ljubljana, Slovenia, October 14-16, 2019, Proceedings 7_, pages 44-55. Springer.
* Tran et al. (2023) Cong Dao Tran, Nbut Hay Pham, Anh Tuan Nguyen, Truong Son Hy, and Tu Vu. 2023. ViDeBERTa: A powerful pre-trained language model for Vietnamese. In _Findings of the Association for Computational Linguistics: EACL 2023_, pages 1071-1078, Dubrovnik, Croatia. Association for Computational Linguistics.
* Tran et al. (2022) Nguyen Luong Tran, Duong Minh Le, and Dat Quoc Nguyen. 2022. Bartpho: Pre-trained sequence-to-sequence models for vietnamese. In _23rd Annual Conference of the International Speech Communication Association, Interspeech 2022, Incheon, Korea, September 18-22, 2022_, pages 1751-1755. ISCA.
* Truong et al. (2021) Thinh Hung Truong, Mai Hoang Dao, and Dat Quoc Nguyen. 2021. COVID-19 named entity recognition for Vietnamese. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human 

Language Technologies_, pages 2146-2153, Online. Association for Computational Linguistics.
* Wolf et al. (2019) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, et al. 2019. Huggingface's transformers: State-of-the-art natural language processing. _ArXiv preprint_, abs/1910.03771.
* Yadav et al. (2020) Hemant Yadav, Sreyan Ghosh, Yi Yu, and Rajiv Ratn Shah. 2020. End-to-end named entity recognition from english speech. In _Interspeech 2020, 21st Annual Conference of the International Speech Communication Association, Virtual Event, Shanghai, China, 25-29 October 2020_, pages 4268-4272. ISCA.

 



###### Contents

* 1 Introduction
* 2 Related Works
* 3 Data
	* 3.1 Data Collection
	* 3.2 Annotation Process
	* 3.3 Data Quality Control
	* 3.4 Data Splitting
	* 3.5 Data Statistics
* 4 Experimental Setups
	* 4.1 Evaluation Metrics
	* 4.2 ASR Models
	* 4.3 NER Models
		* 4.3.1 Seq2seq Training for NER Task
		* 4.3.2 Training Hyperparameters
* 5 Experimental Results
* 6 Error Analysis
	* 6.1 Quantitative
	* 6.2 Qualitative
* 7 Conclusion
* 8 Limitations
* A Annotation Guidelines
* B Discussion about Named-Entity-Error-Rate (NEER)
* B.1 Motivation of NEER
* B.2 Definition of WER
* B.3 Definition of KER
* B.4 Definition of NEER
* B.5 Open questions on NEER
* C Possible Applications
* D Details about Experimental Setups
* D.1 Evaluation Toolkit
* D.2 Modified Evaluation of SLUE toolkit
* E NER Results by Entity Types



## Appendix A Annotation Guidelines

This section describes annotation guidelines for annotators to follow in an attempt to have a unified and consistent gold-standard NER transcript.

**General rules**:

* If 2 or more entities overlap, label the resulting entity as the longest, including overlapping component entities. In other words, a full NE might contain 2 or more sub-NEs. A full NE should be tagged instead of multiple sub-NEs. For example: "bac si xutong khop" (orthopedic doctor) should be tagged as a whole instead of 2 distinct NEs "bac si " (doctor) and "xutong khop" (orthopedic).
* We adhere to the conventional approach of annotating overlapping NE components as a whole, utilizing the BIO encoding scheme. In recent years, research on overlapping and discontinuous NER has introduced alternative annotation frameworks to solve NE overlapping, such as BILOU encoding, which represents "Beginning, Inside, and Last tokens of multi-token chunks, Unit-length chunks, and Outside" [19, 30]. Another approach by Li et al. li2018 introduces a novel span-based model capable of jointly recognizing both overlapping and discontinuous entities. The model operates in two primary stages. First, entity fragments are identified by systematically traversing all possible text spans, enabling the detection of overlapping entities. Second, a relation classification step determines whether a given pair of entity fragments exhibits an overlapping or successive relationship. This approach facilitates the recognition of discontinuous entities while simultaneously verifying overlapping entities.
* Do not assign spaces at the beginning and at the end of entities.
* All words in the ASR transcript are lowercase, without punctuations and special characters. Treat every word as lowercase or uppercase, with or without punctuations and special characters based on the context of each utterance.
* Each utterance should be treated as an independent utterance. The additional context given by other utterances should not influence the annotation of each utterance.

**AGE**:

This entity type describes the age of a person.

* Label the word "tuoi" (age) if applicable. For example: "tuoi truong thanhn" (mature age), "hai bay tuoi" (twenty-seven years old).
* List a range of ages if applicable For example: "hai muoti den ba lam tuoi" (twenty to thirty-five years old), "duoi sau thang tuoi" (under six months old).
* Include adjectives and nouns that might describe how old a person is but don't explicitly describe gender or gender is neutral. For example: "chuta truong thanh" (immature), "ngui gia" (old person), "cu" (sir, old).

**GENDER**:

This entity type describes the gender of a person.

* Include typical entities that are widely understood to describe the gender of a person. For example: "nam" (male), "dan ong" (gentleman), "phu n\({}^{\text{w}}\) (woman).
* Include the titles and pronouns that explicitly describe a gender instead of age. For example: "ogn" (grandfather), "ba" (grandmother), "c6" (aunt), "chi" (uncle).

**OCCUPATION**:

This entity type describes the job of a person.

* Include all jobs that might be both in medical fields and non-medical fields. For example: "khan thinh gia" (audience), "beh hnan" (patient), "ngtoi dan" (citizen), "chuyen gia" (expert).
* Include academic titles and degrees. For example: "thac si" (master degree holder), "tien si" (doctorate), "truong khoa" (dean), "chu rich" (president).
* Include a cluster of words that might describe the specializations of doctors. For example: "bac si chuyen ve roi loan van dong" (doctor who specializes in movement disorders) instead of two distinct entities "bac si" (doctor) and "roi loan vdn long" (movement disorders), "bac si ve parkinson" (parkinson's doctor) instead of two distinct entities "bac si" (doctor) and "parkinson", "bac si chuyen khoa tim mach" (cardiovascular specialist) instead of two distinct entities "bac si" (doctor) and "chuyen khoa tim mach" (cardiovascular)



**LOCATION**:

This entity type describes a location.

* Include continents, countries, regions, cities, and geographical administrative units. For example: "chau au" (europe), "hoa ky" (usa), "stay tang" (tibet), "thanh pho h chi minhn" (ho chi minh city), "tin vinh long" (vinh long province).
* Label words that mean geographical administrative units if applicable. For example: "huyen" (rural district), "quan" (urban district), "duong ph\({}^{\circ}\) (street), "thanh ph\({}^{\circ}\)" (city).
* Include words that might describe public and private sites. For example: "tai nha" (at home), "dong ruong" (farm), "tiem thuoc" (drugstore), "nha may" (factory), "cua hiEU quan ao" (clothing store), "toilet" (toilet).
* Include words that might describe ambient environments. For example: "tai khu ph\({}^{\circ}\)" (in the neighborhood), "tai dia photong" (in local area), "nuoc ngoi" (in foreign countries), "dia ban" (area), "ngoai troi" (outside).
* Include words that might describe medical facilities. For example: "cuhyen khoa tieu haa" (gastrointestinal room) , "icu" (intensive care unit), "tram xa" (clincs), "phong thi nghiem" (laboratory).
* Each level of the administrative unit is a separate entity.
* Do not assign nationality as an entity.
* Locations might be misrecognized as organizations. Do not label places that are not clearly identified or controversial.

**DISEASESYMPTOM**:

This entity type describes a symptom or disease.

* Include the complements of the disease. For example: "bien chung" (side-effect), "chan thutong" (damaged), "bam sinh" (congenital), "di chung" (sequelae), "bi ton thutong" (damaged), "tai phit" (relapse), "dtong tinh" (positive), "benh ly man tnih" (chronic disease), "hoi chung" (syndrome).
* Include a cluster of words that might describe the severity of a disease. For example: "phong cap do ba" (third-degree burn), "svtc de khang kcm" (poor immune system).
* Mental state might also describe mental diseases or their symptoms. For example: "tu f!" (self-deprecation), "tinh trap lo au" (state of anxiety), "met moi ve thin than" (mental fatigue).
* Skin conditions might describe dermatosis or its symptoms. For example: "nam" (melasma), "da d& du" (ouly skin), "da kho" (dry skin), "sam da" (dark skin).
* Genital conditions might describe genital diseases or their symptoms. For example: "co kinkh" (menstruation), "co thai" (pregnant), "day thi som" (early puberty).
* Healthy conditions might help doctors diagnose. For example: "kinh nguyet deu" (regular menstruation).
* Words describing physical status might also speak of symptoms or diseases. For example: "buon ngu" (sleepy), "rung toc" (hair loss), "coin coc" (stunted).
* Words describing children's activities might also speak of pediatric symptoms or diseases. For example: "quay khoc" (fussy), "khong the giao tip" (unable to speak), "cham di" (delay walking).
* Medical techniques or devices might make symptoms and diseases happen. For example: "phau thaut tham my" (cosmetic surgery).

**DRUGCHEMICAL**:

This entity type describes a bio-chemical substance or medicament.

* Extraction of human or animal bodies to serve medical treatment might be referred to as a biochemical substance. For example: "vacxin" (vaccine), "huyt thanh" (blood serum)
* Cosmetics might be referred to as chemical substances. For example: "kem chong hang" (sunscreen), "kern duong am" (moisturizer).
* Food or drink serving medical treatment purposes or as a part of a chemical compound might be referred to as chemical substances. For example: "nam dong tring hahao" (cordyceps), "nhan sam" (ginseng), "nhung hurou" (deer antler)

* Substances extracted from cells or bodies not serving medical purposes might be referred to as bio-chemical substances. For example: "dich tieu hoa" (digestive fluids), "chat noi sinh" (endogenous substances), "mo hoi" (sweat), "ba nhon" (sebum).
* Air might be referred to as chemical substances. For example: "duong khi" (breath air), "oxy" (oxygen).

**FOODDRINK**: This entity type describes food and beverage.

* Include food and drink that might serve nutrient purposes. For example: "sua" (milk), "ngu coc" (cereal).
* Include food and drink that might be harmful to health. For example: "thuco la" (cigarette), "rtqu bia" (alcohol).
* Include words that generally describe food and beverage. For example: "htc pham" (alignment), "tutc an" (food).

**ORGAN**: This entity type describes an anatomical feature, e.g. human organs, biological cells, etc. Annotators should follow general rules.

**PERSONALCARE**: This entity type describes a personal care procedure, e.g. hygiene routines, skin care, daily habits, etc.

* Activities serving the improvement of physical, aesthetic and mental health instead of medical treatment purposes might be referred to as personal care procedures. For example: "an kieng" (diet), "cham soc da" (skin care), "cham soc rang" ( dental care).
* Methods serving self-improvement of speech ability in speech-language pathology might be referred to as personal care. For example: "ttong tac ngon ngu" (language interaction), "huan luyen ngon ngu" (language training).

**DIAGNOSTICS**: This entity type describes a diagnostic procedure, e.g. lab tests, imaging, blood measurement, etc.

* General words describing diagnostic procedures without explicitly mentioning surgery might be referred to as diagnostic produces. For example: "chan doan" (diagnosis), "xet ngbiem" (test).
* Imaging methods might be referred to as diagnostic procedures instead of medical devices or techniques. For example: "mri" (magnetic resonance imaging), "ct" (computed tomography).

**Treatment**: This entity type describes a non-surgical treatment method for diseases, e.g. physical rehabilitation, injection, psychology, etc.

* Words describing methods of using biochemical substances as non-surgical treatment methods might be referred to as treatment methods. For example: "lieu phap hoocomn" (hormone therapy), "dieu tri hoocomn" (horome treatment), "dieu tr i de ba o goc" (stem cell treatment).
* Words describing methods of using invasive techniques as treatment methods might be referred to as treatment methods. For example: "hoa tri" (chemotherapy), "xa tri" (radiotherapy).
* Words describing methods to improve skin conditions for treatment purposes rather than aesthetics might be referred to as treatment methods. For example: "phuc hai da" (skin recovery), "dc che su xuat sac t6" (inhibit pigmentation).

**SURGERY**: This entity type describes a surgical treatment method for diseases, e.g. implants, neurosurgery, invasion, etc.

* Include pre-surgery procedures that might be integral parts of surgeries. For example: "gay me" (anestheisa), "gay te" (anesthetize).
* Include intervention procedures that might be integral parts of dental care. For example: "nh@ rang" (tooth extraction), "implant" (dental implant).
* Include intervention procedures that might be integral parts of pregnancy or genilats. For example: "sinh m0" (caesarean), "clay trainh thai" (contraceptive implant) 

* Include intervention procedures on arteries even though they might be not integral parts of surgery. For example: "truyen mau" (blood transfusion), "truyen nuco bien" (seawater infusion).
* Include neurosurgical procedures that work with brain waves even though they might be minimally invasive. For example: "kich thich nao sau" (dbs or deep brain stimulation).

**MEDDEVICETECHNIQUE**:

This entity type describes a medical device, instrument, bio-material and technique.

* Medical devices and techniques might be confusing. Annotators are strongly recommended to fully annotate CHEM., FnB, ANAT., PC, DX, TX, and SX before engaging TECH.

**UNITCALIBATOR**:

This entity type describes a medical calibration, e.g. number of doses, calories, length, volume, etc.

* Include a cluster of words that both describe the quantity and its unit. Measurements including length, distance, area, weight, heat, velocity, temperature, etc., should be explicitly tagged. For example: "nam milimet" (five millimeters) instead of "nam" (five) or "milimet" (millimeter).
* Complements to the actual quantity describing its approximation should be included. For example: "koang muoi lam phan tram" (about fifteen percent) instead of "muoi lam phan tram" (fifteen percent).
* Include words that generally describe the quantity. For example: "gan du" (close enough), "cao" (high), "rat la lon" (very large).
* Include words that describe trends of quantity. For example: "giam dutoc it nhat" (reduce at least), "mtc do gia tang" (level increases).

**TRANSPORTATION**:

This entity type describes means of transportation or vehicles.

**DATEIME**:

This entity type describes the date and time.

* Include words describing day, week, month, certain named period, season, year, etc.
* Include words describing a time frame. For example: "bay giot" (now), "ve lau ve dai" (in the long run).
* Include words describing the approximate time. For example: "nhanh nhat co the" (as fast as possible), "cang som" (as soon as possible), "tu tti" (gradually).
* Include words describing repetitions. For example: "qinh ky" (periodically).
* Include a cluster of words that both describe time and its complements. For example: "tu thing ba tro di" (from march onwards) instead of 3 distinct entities "tt" (from), "thang ba" (march), and "tro di" (onwards).

 

Discussion about

Named-Entity-Error-Rate (NEER)

### Motivation of NEER

ASR system performance is typically assessed using WER, which represents the ratio of word insertion, substitution, and deletion errors in a transcript to the total number of spoken words. However, various spoken language understanding tasks, such as spoken NER, depend on identifying keywords in transcripts. Moreover, it's essential to recognize that in medical ASR, medical terms carry much higher significance in doctor-patient conversations and should not be treated equally to regular words. KER is often used to evaluate on keywords but is not a directly comparable metric with WER.

The purpose to introduce NEER aims to bridge the gap between WER and KER. However, it is not intended to replace WER or KER as a standard metric for evaluating domain-specific ASR performance. Instead, NEER serves as a complementary metric, facilitating a more in-depth analysis of ASR errors in specific domains, such as the medical field.

### Definition of WER

WER is calculated based on the Levenshtein distance [10], which represents the smallest count of individual edits (insertions, deletions, or substitutions) needed to transform one word into another.

\[WER=\frac{S+D+I}{N}=\frac{S+D+I}{S+D+C} \tag{1}\]

where S is the number of substitutions, D is the number of deletions, I is the number of insertions, C is the number of correct words, and N is the number of words in the reference data \((N=S+D+C)\).

In other words, S is the number of replaced words. D is the number of missed words that are not in ASR hypothesis but are in reference data. I is the number of added words that are in ASR hypothesis but are not in reference data. The alignment between ASR hypothesis and reference data goes from left to right.

### Definition of KER

Like WER, KER is computed using the Levenshtein distance. Each ASR hypothesis is aligned with its corresponding reference data and KER is calculated based on the keyword set.

\[KER=\frac{F+M}{N} \tag{2}\]

where N is the number of keywords in the reference data, F is the number of falsely recognized keywords, M is the number of missed keywords.

The ASR hypothesis often exceeds the length of all keywords in the reference data, and the insertion errors caused by non-keywords may lead to a skewed result in KER. Therefore, no insertion errors are considered while calculating KER.

### Definition of NEER

In KER metric, N is the number of keywords in the reference data. KER could be characterized as the average number of errors per keyword. Nevertheless, the length of keywords may range from 1 to L (where L equals 5 in certain instances such as NER), making the average number of errors per keyword obscure.

In NEER metric, we want to evaluate on keyword-only like KER metric, while also analyzing errors per word like WER metric. Therefore, we change N into the length of keywords (entities), which characterizes the average number of errors per word of keywords.

### Open questions on NEER

We still leave some questions open for future work. First, the analysis of how each type of word error (substitutions, insertions, deletions) influences NER on top of ASR has not been conducted yet. Second, the empirical relationship between WER, KER, NEER, and F1 score - meaning how KER, NEER, and F1 score are affected by a varying range of WERs -- has not been analyzed either 

Possible Applications

In the context of advanced in-context learning capabilities of LLMs and human-level text-to-speech technologies, the dataset, which preserves entity positional labels throughout the translation process, is applicable not only to Vietnamese but also to other languages.

To facilitate dataset translation using LLMs, we employed a concise, brief contextual instruction via GPT-3.5 Turbo, as illustrated below.

In this example, we utilized a straightforward one-shot in-context learning approach, which successfully produced a high-quality translation while preserving all entity positional labels. We hypothesize that employing many-shot learning, extending the context length of instructions, and more advanced LLMs (e.g. GPT-4, GPT-4o) could yield even more enhanced results.

Recent research in text-to-speech (TTS) technology has achieved human-level performance in aspects such as naturalness and emotional expressiveness Tan et al. (2024); Li et al. (2023); Chen et al. (2023). Consequently, the application of TTS to translated NER transcripts could generate realistic audio data, facilitating further research in additional languages.

 

* [25] T. J. Bostek, M. C. K. Sears, and R. A. Zwicky, "A new approach to the determination of the mass of a 

Details about Experimental Setups

### Evaluation Toolkit

We employed seqeval2 framework commonly used as a default evaluation framework by HuggingFace. However, this framework only works for NER on reference text. Therefore, we also employed the F1 score calculation by shon2023b by using the SLUE toolkit3. This F1 score evaluates an unordered list of NE phrase and tag pairs predicted for each sentence. Our proposed modification of SLUE toolkit was also used and presented below.

Footnote 2: [https://github.com/chakki-works/seqeval](https://github.com/chakki-works/seqeval)

Footnote 3: [https://github.com/asappresearch/slue-toolkit](https://github.com/asappresearch/slue-toolkit)

### Modified Evaluation of SLUE toolkit

Following pre-processing, we calculate the evaluation metrics for the ASR-NER SLUE task. This involves computing precision, recall, and F1-score, which provide insights into the model performance at both an individual label level (per entity) and across all labels (overall).

We introduce a "dummy" token strategy to replace the actual NEs. This approach upholds the focus on the classification of entities rather than the extraction of verbatim phrases, which is suitable for cases where ASR errors might skew the recognition of entities in spoken transcripts.

Let's take an example:

* Reference text: "I have a tooth pain"
* BIO encoding of reference text: [0, 0, B-DISEASESYMTOM, I-DISEASESYMTOM]
* ASR output: "Has teeth pain"
* BIO encoding of ASR output: [0, B-DISEASESYMTOM, I-DISEASESYMTOM]

In the SLUE toolkit, the format (NE type, NE) is used to compare reference text and ASR output, e.g. (DISEASESYMTOM, "tooth pain") and (DISEASESYMTOM, "teeth pain"). This format gives an F1 score of 0.0 although entity type is correctly recognized. In our "dummy" token strategy, we modify the format as (NE type, "dummy"), turning reference text and ASR output to (DISAESSYMTOM, "dummy") and (DISEASESYMTOM, "dummy") respectively. The modified format gives a correct F1 score of 1.0.

We compute two types of overall metrics: micro and macro averages. The micro average metrics aggregate the contributions of all classes to compute the average metric, while the macro average computes per-entity type metrics and averages them, without considering the frequency of each entity type. The micro average is therefore influenced by the class distribution and will be dominated by the performance on more frequent entity types. In contrast, macro averages treat all entity types equally, providing a measure of the system's performance across different types of NEs, regardless of their frequency in the dataset.

 

NER Results by Entity Types

Tables 5-24 show the results of NER on reference text by entity types using various pre-trained language models. Tables 25-44 show the results of NER on ASR output by entity types using various pre-trained language models and ASR models.

Figure 1 shows the scatter plot of NER results on reference text by entity types using various pre-trained language models, created by Tables 5-24. Figure 2 shows the scatter plot of NER results on ASR output by entity types using various pre-trained language models, created by Tables 25-44.

 



\begin{table}
\begin{tabular}{|l|c|l|c|c|c|} \hline
**Entity Type** & **NER** & **Eval. Toolkit** & **Prec.** & **Rec.** & **F1** \\ \hline  & BARTpho & Mod. SLUE & 0.70 & 0.79 & 0.74 \\ \cline{2-5}  & BARTpho & seqeval & 0.69 & 0.61 & 0.65 \\ \cline{2-5}  & BARTpho & SLUE & 0.69 & 0.78 & 0.73 \\ \cline{2-5}  & mBART-50 & Mod. SLUE & 0.64 & 0.81 & 0.71 \\ \cline{2-5}  & mBART-50 & seqeval & 0.66 & 0.53 & 0.59 \\ \cline{2-5}  & mBART-50 & SLUE & 0.62 & 0.79 & 0.70 \\ \cline{2-5}  & PhoBERT\_base & Mod. SLUE & 0.75 & 0.81 & 0.78 \\ \cline{2-5}  & PhoBERT\_base & seqeval & 0.76 & 0.62 & 0.68 \\ \cline{2-5}  & PhoBERT\_base & SLUE & 0.74 & 0.80 & 0.77 \\ \cline{2-5}  & PhoBERT\_base-v2 & Mod. SLUE & 0.78 & 0.78 & 0.78 \\ \cline{2-5}  & PhoBERT\_base-v2 & seqeval & 0.76 & 0.66 & 0.71 \\ \cline{2-5}  & PhoBERT\_base-v2 & SLUE & 0.77 & 0.77 & 0.77 \\ \cline{2-5}  & PhoBERT\_large & Mod. SLUE & 0.77 & 0.81 & 0.79 \\ \cline{2-5} AGE & PhoBERT\_large & seqeval & 0.77 & 0.66 & 0.71 \\ \cline{2-5}  & PhoBERT\_large & SLUE & 0.76 & 0.80 & 0.78 \\ \cline{2-5}  & ViDeBERTa\_base & Mod. SLUE & 0.59 & 0.05 & 0.09 \\ \cline{2-5}  & ViDeBERTa\_base & seqeval & 0.03 & 0.29 & 0.05 \\ \cline{2-5}  & ViDeBERTa\_base & SLUE & 0.50 & 0.04 & 0.07 \\ \cline{2-5}  & ViT5\_base & Mod. SLUE & 0.71 & 0.79 & 0.75 \\ \cline{2-5}  & ViT5\_base & seqeval & 0.73 & 0.63 & 0.68 \\ \cline{2-5}  & ViT5\_base & SLUE & 0.69 & 0.77 & 0.73 \\ \cline{2-5}  & XLM-R\_base & Mod. SLUE & 0.69 & 0.79 & 0.73 \\ \cline{2-5}  & XLM-R\_base & seqeval & 0.71 & 0.56 & 0.63 \\ \cline{2-5}  & XLM-R\_base & SLUE & 0.68 & 0.77 & 0.72 \\ \cline{2-5}  & XLM-R\_large & Mod. SLUE & 0.78 & 0.77 & 0.78 \\ \cline{2-5}  & XLM-R\_large & seqeval & 0.75 & 0.67 & 0.71 \\ \cline{2-5}  & XLM-R\_large & SLUE & 0.78 & 0.77 & 0.77 \\ \hline \end{tabular}
\end{table}
Table 5: NER results of **AGE** entity type (in percent) on **reference text** of test set using various pre-trained language models. Metrics shown are Precision, Recall, and overall micro/macro F1 score. Evaluation toolkits used are: seqeval, SLUE and our modified SLUE.





\begin{table}
\begin{tabular}{|l|c|l|c|c|c|} \hline
**Entity Type** & **NER** & **Eval. Toolkit** & **Prec.** & **Rec.** & **F1** \\ \hline  & BARTpho & Mod. SLUE & 0.76 & 0.76 & 0.76 \\ \cline{2-5}  & BARTpho & seqeval & 0.62 & 0.66 & 0.64 \\ \cline{2-5}  & BARTpho & SLUE & 0.74 & 0.75 & 0.75 \\ \cline{2-5}  & mBART-50 & Mod. SLUE & 0.83 & 0.74 & 0.78 \\ \cline{2-5}  & mBART-50 & seqeval & 0.67 & 0.75 & 0.71 \\ \cline{2-5}  & mBART-50 & SLUE & 0.82 & 0.73 & 0.77 \\ \cline{2-5}  & PhoBERT\_base & Mod. SLUE & 0.80 & 0.83 & 0.82 \\ \cline{2-5}  & PhoBERT\_base & seqeval & 0.77 & 0.70 & 0.74 \\ \cline{2-5}  & PhoBERT\_base & SLUE & 0.80 & 0.82 & 0.81 \\ \cline{2-5}  & PhoBERT\_base-v2 & Mod. SLUE & 0.81 & 0.84 & 0.83 \\ \cline{2-5}  & PhoBERT\_base-v2 & seqeval & 0.78 & 0.73 & 0.75 \\ \cline{2-5}  & PhoBERT\_base-v2 & SLUE & 0.81 & 0.83 & 0.82 \\ \cline{2-5}  & PhoBERT\_large & Mod. SLUE & 0.83 & 0.82 & 0.83 \\ \cline{2-5} DATETIME & PhoBERT\_large & seqeval & 0.76 & 0.75 & 0.76 \\ \cline{2-5}  & PhoBERT\_large & SLUE & 0.83 & 0.81 & 0.82 \\ \cline{2-5}  & ViDeBERTa\_base & Mod. SLUE & 0.62 & 0.68 & 0.65 \\ \cline{2-5}  & ViDeBERTa\_base & seqeval & 0.58 & 0.43 & 0.50 \\ \cline{2-5}  & ViDeBERTa\_base & SLUE & 0.60 & 0.65 & 0.62 \\ \cline{2-5}  & ViT5\_base & Mod. SLUE & 0.74 & 0.77 & 0.75 \\ \cline{2-5}  & ViT5\_base & seqeval & 0.69 & 0.67 & 0.68 \\ \cline{2-5}  & ViT5\_base & SLUE & 0.72 & 0.75 & 0.74 \\ \cline{2-5}  & XLM-R\_base & Mod. SLUE & 0.81 & 0.78 & 0.80 \\ \cline{2-5}  & XLM-R\_base & seqeval & 0.72 & 0.70 & 0.71 \\ \cline{2-5}  & XLM-R\_base & SLUE & 0.80 & 0.77 & 0.78 \\ \cline{2-5}  & XLM-R\_large & Mod. SLUE & 0.85 & 0.82 & 0.83 \\ \cline{2-5}  & XLM-R\_large & seqeval & 0.76 & 0.77 & 0.77 \\ \cline{2-5}  & XLM-R\_large & SLUE & 0.84 & 0.81 & 0.82 \\ \hline \end{tabular}
\end{table}
Table 6: NER results of **DATETIME** entity type (in percent) on **reference text** of test set using various pre-trained language models. Metrics shown are Precision, Recall, and overall micro/macro F1 score. Evaluation toolkits used are: seqeval, SLUE and our modified SLUE.





\begin{table}
\begin{tabular}{|c|l|l|r|r|r|} \hline
**Entity Type** & **NER** & **Eval. Toolkit** & **Prec.** & **Rec.** & **F1** \\ \hline  & BARTpho & Mod. SLUE & 0.66 & 0.82 & 0.73 \\ \cline{2-5}  & BARTpho & seqeval & 0.72 & 0.65 & 0.68 \\ \cline{2-5}  & BARTpho & SLUE & 0.65 & 0.82 & 0.73 \\ \cline{2-5}  & mBART-50 & Mod. SLUE & 0.58 & 0.81 & 0.68 \\ \cline{2-5}  & mBART-50 & seqeval & 0.71 & 0.59 & 0.65 \\ \cline{2-5}  & mBART-50 & SLUE & 0.57 & 0.80 & 0.66 \\ \cline{2-5}  & PhoBERT\_base & Mod. SLUE & 0.66 & 0.81 & 0.73 \\ \cline{2-5}  & PhoBERT\_base & seqeval & 0.75 & 0.64 & 0.69 \\ \cline{2-5}  & PhoBERT\_base & SLUE & 0.66 & 0.80 & 0.72 \\ \cline{2-5}  & PhoBERT\_base-v2 & Nod. SLUE & 0.66 & 0.82 & 0.73 \\ \cline{2-5}  & PhoBERT\_base-v2 & seqeval & 0.77 & 0.65 & 0.70 \\ \cline{2-5}  & PhoBERT\_base-v2 & SLUE & 0.66 & 0.82 & 0.73 \\ \cline{2-5}  & PhoBERT\_large & Nod. SLUE & 0.70 & 0.83 & 0.76 \\ \cline{2-5} DIAGNOSTICS & PhoBERT\_large & seqeval & 0.78 & 0.69 & 0.73 \\ \cline{2-5}  & PhoBERT\_large & SLUE & 0.69 & 0.83 & 0.75 \\ \cline{2-5}  & ViDeBERTa\_base & Mod. SLUE & 0.53 & 0.60 & 0.56 \\ \cline{2-5}  & ViDeBERTa\_base & seqeval & 0.55 & 0.39 & 0.45 \\ \cline{2-5}  & ViDeBERTa\_base & SLUE & 0.51 & 0.58 & 0.54 \\ \cline{2-5}  & ViT5\_base & Nod. SLUE & 0.60 & 0.80 & 0.69 \\ \cline{2-5}  & ViT5\_base & seqeval & 0.71 & 0.62 & 0.67 \\ \cline{2-5}  & ViT5\_base & SLUE & 0.59 & 0.78 & 0.67 \\ \cline{2-5}  & XLM-R\_base & Mod. SLUE & 0.61 & 0.82 & 0.70 \\ \cline{2-5}  & XLM-R\_base & seqeval & 0.75 & 0.58 & 0.65 \\ \cline{2-5}  & XLM-R\_base & SLUE & 0.60 & 0.82 & 0.69 \\ \cline{2-5}  & XLM-R\_large & Nod. SLUE & 0.69 & 0.78 & 0.73 \\ \cline{2-5}  & XLM-R\_large & seqeval & 0.78 & 0.69 & 0.73 \\ \cline{2-5}  & XLM-R\_large & SLUE & 0.69 & 0.78 & 0.73 \\ \hline \end{tabular}
\end{table}
Table 7: NER results of **DIAGNOSTICS** entity type (in percent) on **reference text** of test set using various pre-trained language models. Metrics shown are Precision, Recall, and overall micro/macro F1 score. Evaluation toolkits used are: seqeval, SLUE and our modified SLUE.





\begin{table}
\begin{tabular}{|c|l|l|l|l|l|} \hline
**Entity Type** & **NER** & **Eval. Toolkit** & **Prec.** & **Rec.** & **F1** \\ \hline  & BARTPho & Mod. SLUE & 0.75 & 0.81 & 0.78 \\ \cline{2-6}  & BARTPho & seqeval & 0.63 & 0.60 & 0.61 \\ \cline{2-6}  & BARTPho & SLUE & 0.73 & 0.78 & 0.75 \\ \cline{2-6}  & mBART-50 & Mod. SLUE & 0.71 & 0.73 & 0.72 \\ \cline{2-6}  & mBART-50 & seqeval & 0.57 & 0.56 & 0.57 \\ \cline{2-6}  & mBART-50 & SLUE & 0.68 & 0.69 & 0.69 \\ \cline{2-6}  & PhoBERT\_base & Mod. SLUE & 0.79 & 0.83 & 0.81 \\ \cline{2-6}  & PhoBERT\_base & seqeval & 0.70 & 0.60 & 0.65 \\ \cline{2-6}  & PhoBERT\_base & SLUE & 0.77 & 0.80 & 0.78 \\ \cline{2-6}  & PhoBERT\_base-v2 & Mod. SLUE & 0.79 & 0.85 & 0.82 \\ \cline{2-6}  & PhoBERT\_base-v2 & seqeval & 0.73 & 0.61 & 0.66 \\ \cline{2-6}  & PhoBERT\_base-v2 & SLUE & 0.77 & 0.82 & 0.79 \\ \cline{2-6}  & PhoBERT\_large & Mod. SLUE & 0.82 & 0.81 & 0.82 \\ \cline{2-6}  & PhoBERT\_large & seqeval & 0.71 & 0.64 & 0.67 \\ \cline{2-6}  & PhoBERT\_large & SLUE & 0.79 & 0.79 & 0.79 \\ \cline{2-6}  & ViDeBERTa\_base & Mod. SLUE & 0.68 & 0.52 & 0.59 \\ \cline{2-6}  & ViDeBERTa\_base & seqeval & 0.35 & 0.38 & 0.37 \\ \cline{2-6}  & ViDeBERTa\_base & SLUE & 0.62 & 0.47 & 0.54 \\ \cline{2-6}  & ViT5\_base & Mod. SLUE & 0.78 & 0.84 & 0.81 \\ \cline{2-6}  & ViT5\_base & seqeval & 0.73 & 0.67 & 0.70 \\ \cline{2-6}  & ViT5\_base & SLUE & 0.77 & 0.82 & 0.79 \\ \cline{2-6}  & XLM-R\_base & Mod. SLUE & 0.75 & 0.83 & 0.79 \\ \cline{2-6}  & XLM-R\_base & seqeval & 0.69 & 0.56 & 0.61 \\ \cline{2-6}  & XLM-R\_base & SLUE & 0.72 & 0.79 & 0.75 \\ \cline{2-6}  & XLM-R\_large & Mod. SLUE & 0.83 & 0.81 & 0.82 \\ \cline{2-6}  & XLM-R\_large & seqeval & 0.70 & 0.66 & 0.68 \\ \cline{2-6}  & XLM-R\_large & SLUE & 0.81 & 0.79 & 0.80 \\ \hline \end{tabular}
\end{table}
Table 8: NER results of **DISEASESYMTOM** entity type (in percent) on **reference text** of test set using various pre-trained language models. Metrics shown are Precision, Recall, and overall micro/macro F1 score. Evaluation toolkits used are: seqeval, SLUE and our modified SLUE.





\begin{table}
\begin{tabular}{|c|l|l|l|l|l|} \hline
**Entity Type** & **NER** & **Eval. Toolkit** & **Prec.** & **Rec.** & **F1** \\ \hline  & BARTpho & Mod. SLUE & 0.73 & 0.82 & 0.77 \\ \cline{2-6}  & BARTpho & seqeval & 0.75 & 0.70 & 0.73 \\ \cline{2-6}  & BARTpho & SLUE & 0.73 & 0.81 & 0.77 \\ \cline{2-6}  & mBART-50 & Mod. SLUE & 0.79 & 0.69 & 0.74 \\ \cline{2-6}  & mBART-50 & seqeval & 0.71 & 0.76 & 0.73 \\ \cline{2-6}  & mBART-50 & SLUE & 0.79 & 0.69 & 0.74 \\ \cline{2-6}  & PhoBERT\_base & Mod. SLUE & 0.80 & 0.93 & 0.86 \\ \cline{2-6}  & PhoBERT\_base & seqeval & 0.91 & 0.77 & 0.83 \\ \cline{2-6}  & PhoBERT\_base & SLUE & 0.79 & 0.93 & 0.86 \\ \cline{2-6}  & PhoBERT\_base-v2 & Mod. SLUE & 0.83 & 0.93 & 0.88 \\ \cline{2-6}  & PhoBERT\_base-v2 & seqeval & 0.91 & 0.80 & 0.85 \\ \cline{2-6}  & PhoBERT\_base-v2 & SLUE & 0.83 & 0.93 & 0.88 \\ \cline{2-6}  & PhoBERT\_large & Mod. SLUE & 0.74 & 0.93 & 0.82 \\ \cline{2-6} DRUGCHEMICAL & PhoBERT\_large & seqeval & 0.93 & 0.72 & 0.81 \\ \cline{2-6}  & PhoBERT\_large & SLUE & 0.74 & 0.93 & 0.82 \\ \cline{2-6}  & ViDeBERTa\_base & Mod. SLUE & 0.65 & 0.41 & 0.51 \\ \cline{2-6}  & ViDeBERTa\_base & seqeval & 0.31 & 0.41 & 0.35 \\ \cline{2-6}  & ViDeBERTa\_base & SLUE & 0.60 & 0.38 & 0.47 \\ \cline{2-6}  & ViT5\_base & Mod. SLUE & 0.75 & 0.87 & 0.80 \\ \cline{2-6}  & ViT5\_base & seqeval & 0.83 & 0.74 & 0.78 \\ \cline{2-6}  & ViT5\_base & SLUE & 0.75 & 0.86 & 0.80 \\ \cline{2-6}  & XLM-R\_base & Mod. SLUE & 0.81 & 0.74 & 0.77 \\ \cline{2-6}  & XMM-R\_base & seqeval & 0.76 & 0.76 & 0.76 \\ \cline{2-6}  & XMM-R\_base & SLUE & 0.80 & 0.74 & 0.77 \\ \cline{2-6}  & XMM-R\_large & Mod. SLUE & 0.85 & 0.92 & 0.88 \\ \cline{2-6}  & XMM-R\_large & seqeval & 0.91 & 0.79 & 0.85 \\ \cline{2-6}  & XMM-R\_large & SLUE & 0.85 & 0.92 & 0.88 \\ \hline \end{tabular}
\end{table}
Table 9: NER results of **DRUGCHEMICAL** entity type (in percent) on **reference text** of test set using various pre-trained language models. Metrics shown are Precision, Recall, and overall micro/macro F1 score. Evaluation toolkits used are: seqeval, SLUE and our modified SLUE.





\begin{table}
\begin{tabular}{|c|l|l|l|l|l|} \hline
**Entity Type** & **NER** & **Eval. Toolkit** & **Prec.** & **Rec.** & **F1** \\ \hline  & BARTpho & Mod. SLUE & 0.67 & 0.83 & 0.74 \\ \cline{2-5}  & BARTpho & seqeval & 0.69 & 0.59 & 0.64 \\ \cline{2-5}  & BARTpho & SLUE & 0.67 & 0.83 & 0.74 \\ \cline{2-5}  & mBART-50 & Mod. SLUE & 0.56 & 0.67 & 0.61 \\ \cline{2-5}  & mBART-50 & seqeval & 0.57 & 0.52 & 0.54 \\ \cline{2-5}  & mBART-50 & SLUE & 0.56 & 0.67 & 0.61 \\ \cline{2-5}  & PhoBERT\_base & Mod. SLUE & 0.78 & 0.84 & 0.81 \\ \cline{2-5}  & PhoBERT\_base & seqeval & 0.74 & 0.67 & 0.70 \\ \cline{2-5}  & PhoBERT\_base & SLUE & 0.78 & 0.84 & 0.81 \\ \cline{2-5}  & PhoBERT\_base-v2 & Mod. SLUE & 0.82 & 0.89 & 0.85 \\ \cline{2-5}  & PhoBERT\_base-v2 & seqeval & 0.83 & 0.76 & 0.79 \\ \cline{2-5}  & PhoBERT\_base-v2 & SLUE & 0.82 & 0.89 & 0.85 \\ \cline{2-5}  & PhoBERT\_large & Mod. SLUE & 0.80 & 0.91 & 0.85 \\ \cline{2-5} FOODDRINK & PhoBERT\_large & seqeval & 0.85 & 0.75 & 0.80 \\ \cline{2-5}  & PhoBERT\_large & SLUE & 0.80 & 0.91 & 0.85 \\ \cline{2-5}  & ViDeBERTa\_base & Mod. SLUE & 0.22 & 0.38 & 0.28 \\ \cline{2-5}  & ViDeBERTa\_base & seqeval & 0.24 & 0.12 & 0.16 \\ \cline{2-5}  & ViDeBERTa\_base & SLUE & 0.19 & 0.32 & 0.23 \\ \cline{2-5}  & ViT5\_base & Mod. SLUE & 0.70 & 0.86 & 0.77 \\ \cline{2-5}  & ViT5\_base & seqeval & 0.75 & 0.66 & 0.70 \\ \cline{2-5}  & ViT5\_base & SLUE & 0.70 & 0.86 & 0.77 \\ \cline{2-5}  & XLM-R\_base & Mod. SLUE & 0.64 & 0.88 & 0.74 \\ \cline{2-5}  & XLM-R\_base & seqeval & 0.82 & 0.58 & 0.68 \\ \cline{2-5}  & XLM-R\_base & SLUE & 0.62 & 0.85 & 0.72 \\ \cline{2-5}  & XLM-R\_large & Mod. SLUE & 0.88 & 0.81 & 0.84 \\ \cline{2-5}  & XLM-R\_large & seqeval & 0.75 & 0.83 & 0.79 \\ \cline{2-5}  & XLM-R\_large & SLUE & 0.88 & 0.81 & 0.84 \\ \hline \end{tabular}
\end{table}
Table 10: NER results of **FOODDRINK** entity type (in percent) on **reference text** of test set using various pre-trained language models. Metrics shown are Precision, Recall, and overall micro/macro F1 score. Evaluation toolkits used are: seqeval, SLUE and our modified SLUE.





\begin{table}
\begin{tabular}{|c|l|l|l|l|l|} \hline
**Entity Type** & **NER** & **Eval. Toolkit** & **Prec.** & **Rec.** & **F1** \\ \hline  & BARTpho & Mod. SLUE & 0.83 & 0.90 & 0.86 \\ \cline{2-5}  & BARTpho & seqeval & 0.85 & 0.78 & 0.81 \\ \cline{2-5}  & BARTpho & SLUE & 0.82 & 0.90 & 0.86 \\ \cline{2-5}  & mBART-50 & Mod. SLUE & 0.83 & 0.87 & 0.85 \\ \cline{2-5}  & mBART-50 & seqeval & 0.77 & 0.75 & 0.76 \\ \cline{2-5}  & mBART-50 & SLUE & 0.82 & 0.86 & 0.84 \\ \cline{2-5}  & PhoBERT\_base & Mod. SLUE & 0.83 & 0.90 & 0.86 \\ \cline{2-5}  & PhoBERT\_base & seqeval & 0.76 & 0.74 & 0.75 \\ \cline{2-5}  & PhoBERT\_base & SLUE & 0.83 & 0.90 & 0.86 \\ \cline{2-5}  & PhoBERT\_base-v2 & Mod. SLUE & 0.84 & 0.89 & 0.86 \\ \cline{2-5}  & PhoBERT\_base-v2 & seqeval & 0.83 & 0.77 & 0.80 \\ \cline{2-5}  & PhoBERT\_base-v2 & SLUE & 0.84 & 0.89 & 0.86 \\ \cline{2-5}  & PhoBERT\_large & Mod. SLUE & 0.83 & 0.90 & 0.87 \\ \cline{2-5} GENDER & PhoBERT\_large & seqeval & 0.87 & 0.79 & 0.83 \\ \cline{2-5}  & PhoBERT\_large & SLUE & 0.83 & 0.90 & 0.86 \\ \cline{2-5}  & ViDeBERTa\_base & Mod. SLUE & 0.00 & 0.00 & 0.00 \\ \cline{2-5}  & ViDeBERTa\_base & seqeval & 0.00 & 0.00 & 0.00 \\ \cline{2-5}  & ViDeBERTa\_base & SLUE & 0.00 & 0.00 & 0.00 \\ \cline{2-5}  & ViT5\_base & Mod. SLUE & 0.81 & 0.82 & 0.82 \\ \cline{2-5}  & ViT5\_base & seqeval & 0.67 & 0.71 & 0.69 \\ \cline{2-5}  & ViT5\_base & SLUE & 0.81 & 0.82 & 0.81 \\ \cline{2-5}  & XLM-R\_base & Mod. SLUE & 0.79 & 0.72 & 0.75 \\ \cline{2-5}  & XLM-R\_base & seqeval & 0.56 & 0.49 & 0.52 \\ \cline{2-5}  & XLM-R\_base & SLUE & 0.78 & 0.71 & 0.75 \\ \cline{2-5}  & XLM-R\_large & Mod. SLUE & 0.91 & 0.90 & 0.90 \\ \cline{2-5}  & XLM-R\_large & seqeval & 0.82 & 0.79 & 0.80 \\ \cline{2-5}  & XLM-R\_large & SLUE & 0.91 & 0.89 & 0.90 \\ \hline \end{tabular}
\end{table}
Table 11: NER results of **GENDER** entity type (in percent) on **reference text** of test set using various pre-trained language models. Metrics shown are Precision, Recall, and overall micro/macro F1 score. Evaluation toolkits used are: seqeval, SLUE and our modified SLUE.





\begin{table}
\begin{tabular}{|c|l|l|l|l|l|} \hline
**Entity Type** & **NER** & **Eval. Toolkit** & **Prec.** & **Rec.** & **F1** \\ \hline  & BARTpho & Mod. SLUE & 0.77 & 0.88 & 0.82 \\ \cline{2-5}  & BARTpho & seqeval & 0.75 & 0.68 & 0.71 \\ \cline{2-5}  & BARTpho & SLUE & 0.76 & 0.87 & 0.81 \\ \cline{2-5}  & mBART-50 & Mod. SLUE & 0.74 & 0.83 & 0.78 \\ \cline{2-5}  & mBART-50 & seqeval & 0.73 & 0.68 & 0.71 \\ \cline{2-5}  & mBART-50 & SLUE & 0.74 & 0.82 & 0.78 \\ \cline{2-5}  & PhoBERT\_base & Mod. SLUE & 0.75 & 0.92 & 0.82 \\ \cline{2-5}  & PhoBERT\_base & seqeval & 0.81 & 0.63 & 0.71 \\ \cline{2-5}  & PhoBERT\_base & SLUE & 0.74 & 0.91 & 0.81 \\ \cline{2-5}  & PhoBERT\_base-v2 & Mod. SLUE & 0.78 & 0.95 & 0.86 \\ \cline{2-5}  & PhoBERT\_base-v2 & seqeval & 0.86 & 0.69 & 0.77 \\ \cline{2-5}  & PhoBERT\_base-v2 & SLUE & 0.78 & 0.94 & 0.86 \\ \cline{2-5}  & PhoBERT\_large & Mod. SLUE & 0.80 & 0.91 & 0.85 \\ \cline{2-5}   LOCATION & PhoBERT\_large & seqeval & 0.82 & 0.69 & 0.75 \\ \cline{2-5}  & PhoBERT\_large & SLUE & 0.80 & 0.90 & 0.84 \\ \cline{2-5}  & ViDeBERTa\_base & Mod. SLUE & 0.78 & 0.33 & 0.47 \\ \cline{2-5}  & ViDeBERTa\_base & seqeval & 0.32 & 0.60 & 0.42 \\ \cline{2-5}  & ViDeBERTa\_base & SLUE & 0.77 & 0.33 & 0.46 \\ \cline{2-5}  & ViT5\_base & Mod. SLUE & 0.74 & 0.92 & 0.82 \\ \cline{2-5}  & ViT5\_base & seqeval & 0.83 & 0.65 & 0.73 \\ \cline{2-5}  & ViT5\_base & SLUE & 0.73 & 0.92 & 0.81 \\ \cline{2-5}  & XLM-R\_base & Mod. SLUE & 0.75 & 0.84 & 0.79 \\ \cline{2-5}  & XLM-R\_base & seqeval & 0.75 & 0.59 & 0.66 \\ \cline{2-5}  & XLM-R\_base & SLUE & 0.74 & 0.82 & 0.78 \\ \cline{2-5}  & XLM-R\_large & Mod. SLUE & 0.85 & 0.93 & 0.89 \\ \cline{2-5}  & XLM-R\_large & seqeval & 0.87 & 0.75 & 0.81 \\ \cline{2-5}  & XLM-R\_large & SLUE & 0.85 & 0.93 & 0.88 \\ \hline \end{tabular}
\end{table}
Table 12: NER results of **LOCATION** entity type (in percent) on **reference text** of test set using various pre-trained language models. Metrics shown are Precision, Recall, and overall micro/macro F1 score. Evaluation toolkits used are: seqeval, SLUE and our modified SLUE.





\begin{table}
\begin{tabular}{|l|c|l|l|l|l|} \hline
**Entity Type** & **NER** & **Eval. Toolkit** & **Prec.** & **Rec.** & **F1** \\ \hline  & BARTpho & Mod. SLUE & 0.54 & 0.21 & 0.30 \\ \cline{2-6}  & BARTpho & seqeval & 0.16 & 0.39 & 0.23 \\ \cline{2-6}  & BARTpho & SLUE & 0.48 & 0.18 & 0.26 \\ \cline{2-6}  & mBART-50 & Mod. SLUE & 0.45 & 0.09 & 0.15 \\ \cline{2-6}  & mBART-50 & seqeval & 0.08 & 0.35 & 0.13 \\ \cline{2-6}  & mBART-50 & SLUE & 0.36 & 0.07 & 0.12 \\ \cline{2-6}  & PhoBERT\_base & Mod. SLUE & 0.55 & 0.38 & 0.45 \\ \cline{2-6}  & PhoBERT\_base & seqeval & 0.24 & 0.27 & 0.26 \\ \cline{2-6}  & PhoBERT\_base & SLUE & 0.49 & 0.34 & 0.40 \\ \cline{2-6}  & PhoBERT\_base-v2 & Mod. SLUE & 0.59 & 0.38 & 0.46 \\ \cline{2-6}  & PhoBERT\_base-v2 & seqeval & 0.27 & 0.35 & 0.31 \\ \cline{2-6}  & PhoBERT\_base-v2 & SLUE & 0.53 & 0.34 & 0.41 \\ \cline{2-6}  & PhoBERT\_large & Mod. SLUE & 0.61 & 0.30 & 0.40 \\ \cline{2-6} LOCATION & PhoBERT\_large & seqeval & 0.22 & 0.35 & 0.27 \\ \cline{2-6}  & PhoBERT\_large & SLUE & 0.55 & 0.27 & 0.36 \\ \cline{2-6}  & ViDeBERTa\_base & Mod. SLUE & 0.34 & 0.14 & 0.20 \\ \cline{2-6}  & ViDeBERTa\_base & seqeval & 0.06 & 0.09 & 0.07 \\ \cline{2-6}  & ViDeBERTa\_base & SLUE & 0.24 & 0.10 & 0.14 \\ \cline{2-6}  & ViT5\_base & Mod. SLUE & 0.52 & 0.21 & 0.30 \\ \cline{2-6}  & ViT5\_base & seqeval & 0.16 & 0.37 & 0.23 \\ \cline{2-6}  & ViT5\_base & SLUE & 0.47 & 0.19 & 0.27 \\ \cline{2-6}  & XLM-R\_base & Mod. SLUE & 0.48 & 0.27 & 0.34 \\ \cline{2-6}  & XLM-R\_base & seqeval & 0.12 & 0.18 & 0.14 \\ \cline{2-6}  & XLM-R\_base & SLUE & 0.41 & 0.23 & 0.29 \\ \cline{2-6}  & XLM-R\_large & Mod. SLUE & 0.58 & 0.28 & 0.38 \\ \cline{2-6}  & XLM-R\_large & seqeval & 0.20 & 0.32 & 0.25 \\ \cline{2-6}  & XLM-R\_large & SLUE & 0.51 & 0.25 & 0.33 \\ \hline \end{tabular}
\end{table}
Table 13: NER results of **MEDDEVICETECHNIQUE** entity type (in percent) on **reference text** of test set using various pre-trained language models. Metrics shown are Precision, Recall, and overall micro/macro F1 score. Evaluation toolkits used are: seqeval, SLUE and our modified SLUE.





\begin{table}
\begin{tabular}{|c|l|l|l|l|l|} \hline
**Entity Type** & **NER** & **Eval. Toolkit** & **Prec.** & **Rec.** & **F1** \\ \hline  & BARTpho & Mod. SLUE & 0.91 & 0.93 & 0.92 \\ \cline{2-5}  & BARTpho & seqeval & 0.87 & 0.87 & 0.87 \\ \cline{2-5}  & BARTpho & SLUE & 0.91 & 0.92 & 0.92 \\ \cline{2-5}  & mBART-50 & Mod. SLUE & 0.97 & 0.93 & 0.95 \\ \cline{2-5}  & mBART-50 & seqeval & 0.91 & 0.95 & 0.93 \\ \cline{2-5}  & mBART-50 & SLUE & 0.97 & 0.93 & 0.95 \\ \cline{2-5}  & PhoBERT\_base & Mod. SLUE & 0.95 & 0.96 & 0.95 \\ \cline{2-5}  & PhoBERT\_base & seqeval & 0.94 & 0.92 & 0.93 \\ \cline{2-5}  & PhoBERT\_base & SLUE & 0.95 & 0.96 & 0.95 \\ \cline{2-5}  & PhoBERT\_base-v2 & Nod. SLUE & 0.96 & 0.96 & 0.96 \\ \cline{2-5}  & PhoBERT\_base-v2 & seqeval & 0.95 & 0.93 & 0.94 \\ \cline{2-5}  & PhoBERT\_base-v2 & SLUE & 0.96 & 0.96 & 0.96 \\ \cline{2-5}  & PhoBERT\_large & Nod. SLUE & 0.97 & 0.95 & 0.96 \\ \cline{2-5} OCCUPATION & PhoBERT\_large & seqeval & 0.93 & 0.94 & 0.94 \\ \cline{2-5}  & PhoBERT\_large & SLUE & 0.97 & 0.95 & 0.96 \\ \cline{2-5}  & ViDeBERTa\_base & Mod. SLUE & 0.96 & 0.82 & 0.89 \\ \cline{2-5}  & ViDeBERTa\_base & seqeval & 0.81 & 0.91 & 0.86 \\ \cline{2-5}  & ViDeBERTa\_base & SLUE & 0.96 & 0.81 & 0.88 \\ \cline{2-5}  & ViT5\_base & Nod. SLUE & 0.95 & 0.93 & 0.94 \\ \cline{2-5}  & ViT5\_base & seqeval & 0.92 & 0.94 & 0.93 \\ \cline{2-5}  & ViT5\_base & SLUE & 0.95 & 0.93 & 0.94 \\ \cline{2-5}  & XLM-R\_base & Mod. SLUE & 0.88 & 0.96 & 0.92 \\ \cline{2-5}  & XLM-R\_base & seqeval & 0.93 & 0.83 & 0.88 \\ \cline{2-5}  & XLM-R\_base & SLUE & 0.88 & 0.96 & 0.92 \\ \cline{2-5}  & XLM-R\_large & Nod. SLUE & 0.97 & 0.97 & 0.97 \\ \cline{2-5}  & XLM-R\_large & seqeval & 0.95 & 0.95 & 0.95 \\ \cline{2-5}  & XLM-R\_large & SLUE & 0.97 & 0.97 & 0.97 \\ \hline \end{tabular}
\end{table}
Table 14: NER results of **OCCUPATION** entity type (in percent) on **reference text** of test set using various pre-trained language models. Metrics shown are Precision, Recall, and overall micro/macro F1 score. Evaluation toolkits used are: seqeval, SLUE and our modified SLUE.





\begin{table}
\begin{tabular}{|l|c|l|c|c|c|} \hline
**Entity Type** & **NER** & **Eval. Toolkit** & **Prec.** & **Rec.** & **F1** \\ \hline  & BARTpho & Mod. SLUE & 0.72 & 0.80 & 0.76 \\ \cline{2-5}  & BARTpho & seqeval & 0.63 & 0.56 & 0.60 \\ \cline{2-5}  & BARTpho & SLUE & 0.70 & 0.77 & 0.73 \\ \cline{2-5}  & mBART-50 & Mod. SLUE & 0.67 & 0.74 & 0.70 \\ \cline{2-5}  & mBART-50 & seqeval & 0.61 & 0.51 & 0.56 \\ \cline{2-5}  & mBART-50 & SLUE & 0.64 & 0.71 & 0.67 \\ \cline{2-5}  & PhoBERT\_base & Mod. SLUE & 0.74 & 0.86 & 0.79 \\ \cline{2-5}  & PhoBERT\_base & seqeval & 0.69 & 0.55 & 0.61 \\ \cline{2-5}  & PhoBERT\_base & SLUE & 0.71 & 0.83 & 0.77 \\ \cline{2-5}  & PhoBERT\_base-v2 & Mod. SLUE & 0.74 & 0.88 & 0.80 \\ \cline{2-5}  & PhoBERT\_base-v2 & seqeval & 0.70 & 0.55 & 0.62 \\ \cline{2-5}  & PhoBERT\_base-v2 & SLUE & 0.72 & 0.86 & 0.78 \\ \cline{2-5}  & PhoBERT\_large & Mod. SLUE & 0.73 & 0.87 & 0.80 \\ \cline{2-5} ORGAN & PhoBERT\_large & seqeval & 0.71 & 0.55 & 0.62 \\ \cline{2-5}  & PhoBERT\_large & SLUE & 0.71 & 0.85 & 0.77 \\ \cline{2-5}  & ViDeBERTa\_base & Mod. SLUE & 0.53 & 0.51 & 0.52 \\ \cline{2-5}  & ViDeBERTa\_base & seqeval & 0.32 & 0.27 & 0.30 \\ \cline{2-5}  & ViDeBERTa\_base & SLUE & 0.45 & 0.44 & 0.44 \\ \cline{2-5}  & ViT5\_base & Mod. SLUE & 0.71 & 0.85 & 0.78 \\ \cline{2-5}  & ViT5\_base & seqeval & 0.70 & 0.58 & 0.64 \\ \cline{2-5}  & ViT5\_base & SLUE & 0.70 & 0.84 & 0.76 \\ \cline{2-5}  & XLM-R\_base & Mod. SLUE & 0.71 & 0.87 & 0.78 \\ \cline{2-5}  & XLM-R\_base & seqeval & 0.70 & 0.55 & 0.61 \\ \cline{2-5}  & XLM-R\_base & SLUE & 0.69 & 0.85 & 0.76 \\ \cline{2-5}  & XLM-R\_large & Mod. SLUE & 0.77 & 0.87 & 0.82 \\ \cline{2-5}  & XLM-R\_large & seqeval & 0.75 & 0.60 & 0.66 \\ \cline{2-5}  & XLM-R\_large & SLUE & 0.75 & 0.85 & 0.80 \\ \hline \end{tabular}
\end{table}
Table 15: NER results of **ORGAN** entity type (in percent) on **reference text** of test set using various pre-trained language models. Metrics shown are Precision, Recall, and overall micro/macro F1 score. Evaluation toolkits used are: seqeval, SLUE and our modified SLUE.





\begin{table}
\begin{tabular}{|c|l|l|l|l|l|} \hline
**Entity Type** & **NER** & **Eval. Toolkit** & **Prec.** & **Rec.** & **F1** \\ \hline  & BARTrho & Mod. SLUE & 0.73 & 0.54 & 0.62 \\ \cline{2-6}  & BARTrho & seqeval & 0.47 & 0.68 & 0.56 \\ \cline{2-6}  & BARTPho & SLUE & 0.71 & 0.53 & 0.61 \\ \cline{2-6}  & mBART-50 & Mod. SLUE & 0.77 & 0.31 & 0.44 \\ \cline{2-6}  & mBART-50 & seqeval & 0.28 & 0.70 & 0.40 \\ \cline{2-6}  & mBART-50 & SLUE & 0.77 & 0.31 & 0.44 \\ \cline{2-6}  & PhoBERT\_base & Mod. SLUE & 0.71 & 0.57 & 0.63 \\ \cline{2-6}  & PhoBERT\_base & seqval & 0.51 & 0.56 & 0.53 \\ \cline{2-6}  & PhoBERT\_base & SLUE & 0.70 & 0.56 & 0.62 \\ \cline{2-6}  & PhoBERT\_base-v2 & Mod. SLUE & 0.76 & 0.71 & 0.73 \\ \cline{2-6}  & PhoBERT\_base-v2 & seqeval & 0.59 & 0.60 & 0.60 \\ \cline{2-6}  & PhoBERT\_base-v2 & SLUE & 0.74 & 0.70 & 0.72 \\ \cline{2-6}  & PhoBERT\_base-v2 & Mod. SLUE & 0.71 & 0.65 & 0.68 \\ \cline{2-6} ORGANIZATION & PhoBERT\_large & seqeval & 0.59 & 0.50 & 0.54 \\ \cline{2-6}  & PhoBERT\_large & SLUE & 0.71 & 0.65 & 0.68 \\ \cline{2-6}  & ViDeBERTa\_base & Mod. SLUE & 0.00 & 0.00 & 0.00 \\ \cline{2-6}  & ViDeBERTa\_base & seqeval & 0.00 & 0.00 & 0.00 \\ \cline{2-6}  & ViDeBERTa\_base & SLUE & 0.00 & 0.00 & 0.00 \\ \cline{2-6}  & ViT5\_base & Mod. SLUE & 0.94 & 0.36 & 0.52 \\ \cline{2-6}  & ViT5\_base & seqeval & 0.34 & 0.91 & 0.50 \\ \cline{2-6}  & ViT5\_base & SLUE & 0.94 & 0.36 & 0.52 \\ \cline{2-6}  & XLM-R\_base & Mod. SLUE & 0.59 & 0.31 & 0.41 \\ \cline{2-6}  & XLM-R\_base & seqeval & 0.08 & 0.12 & 0.10 \\ \cline{2-6}  & XLM-R\_base & SLUE & 0.56 & 0.29 & 0.39 \\ \cline{2-6}  & XMM-R\_large & Mod. SLUE & 0.82 & 0.66 & 0.73 \\ \cline{2-6}  & XMM-R\_large & seqeval & 0.61 & 0.70 & 0.65 \\ \cline{2-6}  & XLM-R\_large & SLUE & 0.80 & 0.64 & 0.71 \\ \hline \end{tabular}
\end{table}
Table 16: NER results of **ORGANIZATION** entity type (in percent) on **reference text** of test set using various pre-trained language models. Metrics shown are Precision, Recall, and overall micro/macro F1 score. Evaluation toolkits used are: seqeval, SLUE and our modified SLUE.





\begin{table}
\begin{tabular}{|c|l|l|l|l|l|} \hline
**Entity Type** & **NER** & **Eval. Toolkit** & **Prec.** & **Rec.** & **F1** \\ \hline  & BARTpho & Mod. SLUE & 0.17 & 0.77 & 0.28 \\ \cline{2-5}  & BARTpho & seqeval & 0.75 & 0.21 & 0.32 \\ \cline{2-5}  & BARTpho & SLUE & 0.17 & 0.77 & 0.28 \\ \cline{2-5}  & mBART-50 & Mod. SLUE & 0.23 & 0.80 & 0.36 \\ \cline{2-5}  & mBART-50 & seqeval & 0.76 & 0.26 & 0.39 \\ \cline{2-5}  & mBART-50 & SLUE & 0.23 & 0.80 & 0.36 \\ \cline{2-5}  & PhoBERT\_base & Mod. SLUE & 0.18 & 0.87 & 0.30 \\ \cline{2-5}  & PhoBERT\_base & seqval & 0.85 & 0.20 & 0.33 \\ \cline{2-5}  & PhoBERT\_base & SLUE & 0.18 & 0.85 & 0.30 \\ \cline{2-5}  & PhoBERT\_base-v2 & Mod. SLUE & 0.16 & 0.85 & 0.27 \\ \cline{2-5}  & PhoBERT\_base-v2 & seqeval & 0.85 & 0.18 & 0.30 \\ \cline{2-5}  & PhoBERT\_base-v2 & SLUE & 0.16 & 0.84 & 0.27 \\ \cline{2-5}  & PhoBERT\_large & Mod. SLUE & 0.19 & 0.83 & 0.31 \\ \cline{2-5} PERSONALCARE & PhoBERT\_large & seqeval & 0.80 & 0.21 & 0.33 \\ \cline{2-5}  & PhoBERT\_large & SLUE & 0.19 & 0.82 & 0.31 \\ \cline{2-5}  & ViDeBERTa\_base & Mod. SLUE & 0.17 & 0.76 & 0.28 \\ \cline{2-5}  & ViDeBERTa\_base & seqeval & 0.68 & 0.14 & 0.23 \\ \cline{2-5}  & ViDeBERTa\_base & SLUE & 0.17 & 0.75 & 0.28 \\ \cline{2-5}  & ViT5\_base & Mod. SLUE & 0.17 & 0.79 & 0.28 \\ \cline{2-5}  & ViT5\_base & seqeval & 0.75 & 0.18 & 0.29 \\ \cline{2-5}  & ViT5\_base & SLUE & 0.17 & 0.78 & 0.27 \\ \cline{2-5}  & XLM-R\_base & Mod. SLUE & 0.21 & 0.83 & 0.33 \\ \cline{2-5}  & XLM-R\_base & seqeval & 0.77 & 0.21 & 0.33 \\ \cline{2-5}  & XLM-R\_base & SLUE & 0.20 & 0.81 & 0.32 \\ \cline{2-5}  & XLM-R\_large & Mod. SLUE & 0.22 & 0.87 & 0.36 \\ \cline{2-5}  & XLM-R\_large & seqeval & 0.84 & 0.24 & 0.38 \\ \cline{2-5}  & XLM-R\_large & SLUE & 0.22 & 0.86 & 0.35 \\ \hline \end{tabular}
\end{table}
Table 17: NER results of **PERSONALCARE** entity type (in percent) on **reference text** of test set using various pre-trained language models. Metrics shown are Precision, Recall, and overall micro/macro F1 score. Evaluation toolkits used are: seqeval, SLUE and our modified SLUE.





\begin{table}
\begin{tabular}{|c|l|l|l|l|l|} \hline
**Entity Type** & **NER** & **Eval. Toolkit** & **Prec.** & **Rec.** & **F1** \\ \hline  & BARTpho & Mod. SLUE & 0.01 & 0.15 & 0.01 \\ \cline{2-6}  & BARTpho & seqeval & 0.21 & 0.01 & 0.02 \\ \cline{2-6}  & BARTpho & SLUE & 0.01 & 0.15 & 0.01 \\ \cline{2-6}  & mBART-50 & Mod. SLUE & 0.02 & 0.45 & 0.04 \\ \cline{2-6}  & mBART-50 & seqeval & 0.42 & 0.02 & 0.04 \\ \cline{2-6}  & mBART-50 & SLUE & 0.02 & 0.45 & 0.04 \\ \cline{2-6}  & PhoBERT\_base & Mod. SLUE & 0.03 & 0.75 & 0.06 \\ \cline{2-6}  & PhoBERT\_base & seqeval & 0.56 & 0.02 & 0.05 \\ \cline{2-6}  & PhoBERT\_base & SLUE & 0.03 & 0.69 & 0.06 \\ \cline{2-6}  & PhoBERT\_base-v2 & Mod. SLUE & 0.02 & 0.44 & 0.04 \\ \cline{2-6}  & PhoBERT\_base-v2 & seqeval & 0.33 & 0.02 & 0.03 \\ \cline{2-6}  & PhoBERT\_base-v2 & SLUE & 0.02 & 0.39 & 0.03 \\ \cline{2-6}  & PhoBERT\_large & Mod. SLUE & 0.03 & 0.78 & 0.06 \\ \cline{2-6} PREVENTIVEMED & PhoBERT\_large & seqval & 0.72 & 0.03 & 0.06 \\ \cline{2-6}  & PhoBERT\_large & SLUE & 0.03 & 0.75 & 0.06 \\ \cline{2-6}  & ViDeBERTa\_base & Mod. SLUE & 0.00 & 0.06 & 0.01 \\ \cline{2-6}  & ViDeBERTa\_base & seqeval & 0.00 & 0.00 & 0.00 \\ \cline{2-6}  & ViDeBERTa\_base & SLUE & 0.00 & 0.00 & 0.00 \\ \cline{2-6}  & ViT5\_base & Mod. SLUE & 0.02 & 0.39 & 0.04 \\ \cline{2-6}  & ViT5\_base & seqeval & 0.39 & 0.03 & 0.05 \\ \cline{2-6}  & ViT5\_base & SLUE & 0.02 & 0.35 & 0.04 \\ \cline{2-6}  & XLM-R\_base & Mod. SLUE & 0.01 & 0.14 & 0.02 \\ \cline{2-6}  & XLM-R\_base & seqeval & 0.00 & 0.00 & 0.00 \\ \cline{2-6}  & XLM-R\_base & SLUE & 0.01 & 0.08 & 0.01 \\ \cline{2-6}  & XLM-R\_large & Mod. SLUE & 0.03 & 0.78 & 0.06 \\ \cline{2-6}  & XLM-R\_large & seqeval & 0.72 & 0.03 & 0.05 \\ \cline{2-6}  & XLM-R\_large & SLUE & 0.03 & 0.78 & 0.06 \\ \hline \end{tabular}
\end{table}
Table 18: NER results of **PREVENTIVEMD** entity type (in percent) on **reference text** of test set using various pre-trained language models. Metrics shown are Precision, Recall, and overall micro/macro F1 score. Evaluation toolkits used are: seqeval, SLUE and our modified SLUE.





\begin{table}
\begin{tabular}{|l|c|l|c|c|c|} \hline
**Entity Type** & **NER** & **Eval. Toolkit** & **Prec.** & **Rec.** & **F1** \\ \hline  & BARTpho & Mod. SLUE & 0.57 & 0.57 & 0.57 \\ \cline{2-6}  & BARTpho & seqeval & 0.48 & 0.55 & 0.51 \\ \cline{2-6}  & BARTpho & SLUE & 0.57 & 0.56 & 0.57 \\ \cline{2-6}  & mBART-50 & Mod. SLUE & 0.67 & 0.38 & 0.48 \\ \cline{2-6}  & mBART-50 & seqeval & 0.29 & 0.55 & 0.38 \\ \cline{2-6}  & mBART-50 & SLUE & 0.64 & 0.37 & 0.47 \\ \cline{2-6}  & PhoBERT\_base & Mod. SLUE & 0.70 & 0.48 & 0.56 \\ \cline{2-6}  & PhoBERT\_base & seqeval & 0.36 & 0.50 & 0.42 \\ \cline{2-6}  & PhoBERT\_base & SLUE & 0.68 & 0.46 & 0.55 \\ \cline{2-6}  & PhoBERT\_base-v2 & Mod. SLUE & 0.79 & 0.66 & 0.72 \\ \cline{2-6}  & PhoBERT\_base-v2 & seqeval & 0.62 & 0.59 & 0.61 \\ \cline{2-6}  & PhoBERT\_base-v2 & SLUE & 0.78 & 0.65 & 0.71 \\ \cline{2-6}  & PhoBERT\_large & Mod. SLUE & 0.85 & 0.46 & 0.59 \\ \cline{2-6}  & PhoBERT\_large & seqeval & 0.38 & 0.66 & 0.49 \\ \cline{2-6}  & PhoBERT\_large & SLUE & 0.85 & 0.45 & 0.59 \\ \cline{2-6}  & ViDeBERTa\_base & Mod. SLUE & 0.78 & 0.32 & 0.46 \\ \cline{2-6}  & ViDeBERTa\_base & seqeval & 0.24 & 0.45 & 0.32 \\ \cline{2-6}  & ViDeBERTa\_base & SLUE & 0.76 & 0.31 & 0.44 \\ \cline{2-6}  & ViT5\_base & Mod. SLUE & 0.61 & 0.69 & 0.65 \\ \cline{2-6}  & ViT5\_base & seqeval & 0.55 & 0.51 & 0.52 \\ \cline{2-6}  & ViT5\_base & SLUE & 0.60 & 0.69 & 0.64 \\ \cline{2-6}  & XLM-R\_base & Mod. SLUE & 0.66 & 0.67 & 0.66 \\ \cline{2-6}  & XLM-R\_base & seqeval & 0.59 & 0.46 & 0.52 \\ \cline{2-6}  & XLM-R\_base & SLUE & 0.66 & 0.66 & 0.66 \\ \cline{2-6}  & XLM-R\_large & Mod. SLUE & 0.80 & 0.46 & 0.59 \\ \cline{2-6}  & XLM-R\_large & seqeval & 0.39 & 0.65 & 0.49 \\ \cline{2-6}  & XLM-R\_large & SLUE & 0.80 & 0.46 & 0.58 \\ \hline \end{tabular}
\end{table}
Table 19: NER results of **SURGERY** entity type (in percent) on **reference text** of test set using various pre-trained language models. Metrics shown are Precision, Recall, and overall micro/macro F1 score. Evaluation toolkits used are: seqeval, SLUE and our modified SLUE.





\begin{table}
\begin{tabular}{|c|l|l|l|l|l|} \hline
**Entity Type** & **NER** & **Eval. Toolkit** & **Prec.** & **Rec.** & **F1** \\ \hline  & BARTpho & Mod. SLUE & 0.98 & 0.96 & 0.97 \\ \cline{2-5}  & BARTpho & seqeval & 0.93 & 0.93 & 0.93 \\ \cline{2-5}  & BARTpho & SLUE & 0.98 & 0.96 & 0.97 \\ \cline{2-5}  & mBART-50 & Mod. SLUE & 1,00 & 0.70 & 0.82 \\ \cline{2-5}  & mBART-50 & seqeval & 0.67 & 0.95 & 0.78 \\ \cline{2-5}  & mBART-50 & SLUE & 1,00 & 0.70 & 0.82 \\ \cline{2-5}  & PhoBERT\_base & Mod. SLUE & 1,00 & 0.90 & 0.95 \\ \cline{2-5}  & PhoBERT\_base & seqeval & 0.85 & 0.92 & 0.88 \\ \cline{2-5}  & PhoBERT\_base & SLUE & 1,00 & 0.90 & 0.95 \\ \cline{2-5}  & PhoBERT\_base-v2 & Mod. SLUE & 1,00 & 0.69 & 0.82 \\ \cline{2-5}  & PhoBERT\_base-v2 & seqeval & 0.59 & 0.80 & 0.68 \\ \cline{2-5}  & PhoBERT\_base-v2 & SLUE & 1,00 & 0.69 & 0.82 \\ \cline{2-5}  & PhoBERT\_large & Mod. SLUE & 0.95 & 0.91 & 0.93 \\ \cline{2-5} TRANSPORTATION & PhoBERT\_large & seqeval & 0.89 & 0.92 & 0.91 \\ \cline{2-5}  & PhoBERT\_large & SLUE & 0.95 & 0.91 & 0.93 \\ \cline{2-5}  & ViDeBERTa\_base & Mod. SLUE & 0.00 & 0.00 & 0.00 \\ \cline{2-5}  & ViDeBERTa\_base & seqeval & 0.00 & 0.00 & 0.00 \\ \cline{2-5}  & ViDeBERTa\_base & SLUE & 0.00 & 0.00 & 0.00 \\ \cline{2-5}  & ViT5\_base & Mod. SLUE & 0.00 & 0.00 & 0.00 \\ \cline{2-5}  & ViT5\_base & seqeval & 0.00 & 0.00 & 0.00 \\ \cline{2-5}  & ViT5\_base & SLUE & 0.00 & 0.00 & 0.00 \\ \cline{2-5}  & XLM-R\_base & Mod. SLUE & 0.00 & 0.00 & 0.00 \\ \cline{2-5}  & XLM-R\_base & seqeval & 0.00 & 0.00 & 0.00 \\ \cline{2-5}  & XLM-R\_large & SLUE & 0.00 & 0.00 & 0.00 \\ \cline{2-5}  & XLM-R\_base & Mod. SLUE & 1,00 & 0.72 & 0.84 \\ \cline{2-5}  & XLM-R\_large & seqeval & 0.63 & 0.81 & 0.71 \\ \cline{2-5}  & XLM-R\_large & SLUE & 1,00 & 0.72 & 0.84 \\ \hline \end{tabular}
\end{table}
Table 20: NER results of **TRANSPORTATION** entity type (in percent) on **reference text** of test set using various pre-trained language models. Metrics shown are Precision, Recall, and overall micro/macro F1 score. Evaluation toolkits used are: seqeval, SLUE and our modified SLUE.





\begin{table}
\begin{tabular}{|c|l|l|l|l|l|} \hline
**Entity Type** & **NER** & **Eval. Toolkit** & **Prec.** & **Rec.** & **F1** \\ \hline  & BARTpho & Mod. SLUE & 0.63 & 0.82 & 0.71 \\ \cline{2-5}  & BARTpho & seqeval & 0.75 & 0.57 & 0.65 \\ \cline{2-5}  & BARTpho & SLUE & 0.63 & 0.82 & 0.71 \\ \cline{2-5}  & mBART-50 & Mod. SLUE & 0.66 & 0.83 & 0.74 \\ \cline{2-5}  & mBART-50 & seqeval & 0.81 & 0.61 & 0.70 \\ \cline{2-5}  & mBART-50 & SLUE & 0.66 & 0.83 & 0.74 \\ \cline{2-5}  & PhoBERT\_base & Mod. SLUE & 0.66 & 0.88 & 0.75 \\ \cline{2-5}  & PhoBERT\_base & seqeval & 0.86 & 0.60 & 0.71 \\ \cline{2-5}  & PhoBERT\_base & SLUE & 0.66 & 0.88 & 0.75 \\ \cline{2-5}  & PhoBERT\_base-v2 & Mod. SLUE & 0.62 & 0.88 & 0.73 \\ \cline{2-5}  & PhoBERT\_base-v2 & seqeval & 0.87 & 0.56 & 0.68 \\ \cline{2-5}  & PhoBERT\_base-v2 & SLUE & 0.62 & 0.88 & 0.72 \\ \cline{2-5}  & PhoBERT\_large & Mod. SLUE & 0.59 & 0.87 & 0.70 \\ \cline{2-5} TREATMENT & PhoBERT\_large & seqeval & 0.86 & 0.53 & 0.65 \\ \cline{2-5}  & PhoBERT\_large & SLUE & 0.59 & 0.87 & 0.70 \\ \cline{2-5}  & ViDeBERTa\_base & Mod. SLUE & 0.62 & 0.86 & 0.72 \\ \cline{2-5}  & ViDeBERTa\_base & seqeval & 0.84 & 0.52 & 0.64 \\ \cline{2-5}  & ViDeBERTa\_base & SLUE & 0.62 & 0.86 & 0.72 \\ \cline{2-5}  & ViT5\_base & Mod. SLUE & 0.46 & 0.84 & 0.60 \\ \cline{2-5}  & ViT5\_base & seqeval & 0.81 & 0.41 & 0.55 \\ \cline{2-5}  & ViT5\_base & SLUE & 0.46 & 0.84 & 0.60 \\ \cline{2-5}  & XLM-R\_base & Mod. SLUE & 0.48 & 0.86 & 0.62 \\ \cline{2-5}  & XLM-R\_base & seqeval & 0.84 & 0.40 & 0.54 \\ \cline{2-5}  & XLM-R\_base & SLUE & 0.48 & 0.86 & 0.62 \\ \cline{2-5}  & XLM-R\_large & Mod. SLUE & 0.59 & 0.89 & 0.71 \\ \cline{2-5}  & XLM-R\_large & seqeval & 0.89 & 0.54 & 0.67 \\ \cline{2-5}  & XLM-R\_large & SLUE & 0.59 & 0.89 & 0.71 \\ \hline \end{tabular}
\end{table}
Table 21: NER results of **TREATMENT** entity type (in percent) on **reference text** of test set using various pre-trained language models. Metrics shown are Precision, Recall, and overall micro/macro F1 score. Evaluation toolkits used are: seqeval, SLUE and our modified SLUE.





\begin{table}
\begin{tabular}{|c|l|l|l|l|l|} \hline
**Entity Type** & **NER** & **Eval. Toolkit** & **Prec.** & **Rec.** & **F1** \\ \hline  & BARTpho & Mod. SLUE & 0.42 & 0.65 & 0.51 \\ \cline{2-5}  & BARTpho & seqeval & 0.52 & 0.31 & 0.39 \\ \cline{2-5}  & BARTpho & SLUE & 0.41 & 0.63 & 0.50 \\ \cline{2-5}  & mBART-50 & Mod. SLUE & 0.38 & 0.58 & 0.46 \\ \cline{2-5}  & mBART-50 & seqeval & 0.45 & 0.26 & 0.33 \\ \cline{2-5}  & mBART-50 & SLUE & 0.37 & 0.56 & 0.44 \\ \cline{2-5}  & PhoBERT\_base & Mod. SLUE & 0.44 & 0.73 & 0.55 \\ \cline{2-5}  & PhoBERT\_base & seqeval & 0.61 & 0.30 & 0.41 \\ \cline{2-5}  & PhoBERT\_base & SLUE & 0.43 & 0.71 & 0.53 \\ \cline{2-5}  & PhoBERT\_base-v2 & Mod. SLUE & 0.46 & 0.74 & 0.57 \\ \cline{2-5}  & PhoBERT\_base-v2 & seqeval & 0.61 & 0.33 & 0.43 \\ \cline{2-5}  & PhoBERT\_base-v2 & SLUE & 0.45 & 0.72 & 0.55 \\ \cline{2-5}  & PhoBERT\_large & Mod. SLUE & 0.48 & 0.73 & 0.58 \\ \cline{2-5} UNITCALIBRATOR & PhoBERT\_large & seqeval & 0.60 & 0.34 & 0.43 \\ \cline{2-5}  & PhoBERT\_large & SLUE & 0.47 & 0.71 & 0.56 \\ \cline{2-5}  & ViDeBERTa\_base & Mod. SLUE & 0.32 & 0.44 & 0.37 \\ \cline{2-5}  & ViDeBERTa\_base & seqeval & 0.20 & 0.10 & 0.14 \\ \cline{2-5}  & ViDeBERTa\_base & SLUE & 0.28 & 0.39 & 0.33 \\ \cline{2-5}  & ViT5\_base & Mod. SLUE & 0.34 & 0.63 & 0.44 \\ \cline{2-5}  & ViT5\_base & seqeval & 0.50 & 0.24 & 0.32 \\ \cline{2-5}  & ViT5\_base & SLUE & 0.33 & 0.62 & 0.43 \\ \cline{2-5}  & XLM-R\_base & Mod. SLUE & 0.44 & 0.72 & 0.54 \\ \cline{2-5}  & XLM-R\_base & seqeval & 0.54 & 0.27 & 0.36 \\ \cline{2-5}  & XLM-R\_base & SLUE & 0.42 & 0.69 & 0.52 \\ \cline{2-5}  & XLM-R\_large & Mod. SLUE & 0.50 & 0.75 & 0.60 \\ \cline{2-5}  & XLM-R\_large & seqeval & 0.65 & 0.37 & 0.47 \\ \cline{2-5}  & XLM-R\_large & SLUE & 0.49 & 0.74 & 0.59 \\ \hline \end{tabular}
\end{table}
Table 22: NER results of **UNITCALIBRATOR** entity type (in percent) on **reference text** of test set using various pre-trained language models. Metrics shown are Precision, Recall, and overall micro/macro F1 score. Evaluation toolkits used are: seqeval, SLUE and our modified SLUE.





\begin{table}
\begin{tabular}{|c|l|l|l|l|l|} \hline
**Entity Type** & **NER** & **Eval. Toolkit** & **Prec.** & **Rec.** & **F1** \\ \hline  & BARTpho & Mod. SLUE & 0.64 & 0.72 & 0.66 \\ \cline{2-5}  & BARTpho & seqeval & 0.64 & 0.58 & 0.59 \\ \cline{2-5}  & BARTpho & SLUE & 0.63 & 0.71 & 0.65 \\ \cline{2-5}  & mBART-50 & Mod. SLUE & 0.64 & 0.66 & 0.61 \\ \cline{2-5}  & mBART-50 & seqeval & 0.59 & 0.57 & 0.55 \\ \cline{2-5}  & mBART-50 & SLUE & 0.63 & 0.65 & 0.60 \\ \cline{2-5}  & PhoBERT\_base & Mod. SLUE & 0.67 & 0.79 & 0.69 \\ \cline{2-5}  & PhoBERT\_base & seqeval & 0.70 & 0.57 & 0.60 \\ \cline{2-5}  & PhoBERT\_base & SLUE & 0.66 & 0.78 & 0.68 \\ \cline{2-5}  & PhoBERT\_base-v2 & Mod. SLUE & 0.69 & 0.79 & 0.71 \\ \cline{2-5}  & PhoBERT\_base-v2 & seqeval & 0.71 & 0.59 & 0.62 \\ \cline{2-5}  & PhoBERT\_base-v2 & SLUE & 0.68 & 0.77 & 0.70 \\ \cline{2-5}  & PhoBERT\_large & Mod. SLUE & 0.69 & 0.79 & 0.70 \\ \cline{2-5} Overall Macro & PhoBERT\_large & seqeval & 0.73 & 0.60 & 0.63 \\ \cline{2-5}  & PhoBERT\_large & SLUE & 0.68 & 0.78 & 0.69 \\ \cline{2-5}  & ViDeBERTa\_base & Mod. SLUE & 0.43 & 0.38 & 0.37 \\ \cline{2-5}  & ViDeBERTa\_base & seqeval & 0.31 & 0.28 & 0.27 \\ \cline{2-5}  & ViDeBERTa\_base & SLUE & 0.40 & 0.36 & 0.34 \\ \cline{2-5}  & ViT5\_base & Mod. SLUE & 0.59 & 0.69 & 0.60 \\ \cline{2-5}  & ViT5\_base & seqeval & 0.61 & 0.53 & 0.54 \\ \cline{2-5}  & ViT5\_base & SLUE & 0.58 & 0.68 & 0.59 \\ \cline{2-5}  & XLM-R\_base & Mod. SLUE & 0.57 & 0.67 & 0.59 \\ \cline{2-5}  & XLM-R\_base & seqeval & 0.57 & 0.44 & 0.48 \\ \cline{2-5}  & XLM-R\_base & SLUE & 0.56 & 0.65 & 0.58 \\ \cline{2-5}  & XLM-R\_large & Mod. SLUE & 0.72 & 0.78 & 0.71 \\ \cline{2-5}  & XLM-R\_large & seqeval & 0.72 & 0.62 & 0.63 \\ \cline{2-5}  & XLM-R\_large & SLUE & 0.71 & 0.77 & 0.70 \\ \hline \end{tabular}
\end{table}
Table 23: NER results of **Overall Macro** (in percent) on **reference text** of test set using various pre-trained language models. Metrics shown are Precision, Recall, and overall micro/macro F1 score. Evaluation toolkits used are: seqeval, SLUE and our modified SLUE.





\begin{table}
\begin{tabular}{|c|l|l|l|l|l|} \hline
**Entity Type** & **NER** & **Eval. Toolkit** & **Prec.** & **Rec.** & **F1** \\ \hline  & BARTpho & Mod. SLUE & 0.66 & 0.74 & 0.70 \\ \cline{2-5}  & BARTPho & seqeval & 0.64 & 0.58 & 0.61 \\ \cline{2-5}  & BARTpho & SLUE & 0.64 & 0.73 & 0.68 \\ \cline{2-5}  & mBART-50 & Mod. SLUE & 0.65 & 0.68 & 0.67 \\ \cline{2-5}  & mBART-50 & seqeval & 0.60 & 0.57 & 0.59 \\ \cline{2-5}  & mBART-50 & SLUE & 0.64 & 0.66 & 0.65 \\ \cline{2-5}  & PhoBERT\_base & Mod. SLUE & 0.69 & 0.79 & 0.74 \\ \cline{2-5}  & PhoBERT\_base & seqeval & 0.71 & 0.57 & 0.63 \\ \cline{2-5}  & PhoBERT\_base & SLUE & 0.67 & 0.78 & 0.72 \\ \cline{2-5}  & PhoBERT\_base-v2 & Mod. SLUE & 0.70 & 0.81 & 0.75 \\ \cline{2-5}  & PhoBERT\_base-v2 & seqeval & 0.74 & 0.59 & 0.66 \\ \cline{2-5}  & PhoBERT\_base-v2 & SLUE & 0.68 & 0.79 & 0.74 \\ \cline{2-5}  & PhoBERT\_large & Mod. SLUE & 0.70 & 0.79 & 0.74 \\ \cline{2-5} Overall Micro & PhoBERT\_large & seqeval & 0.73 & 0.60 & 0.66 \\ \cline{2-5}  & PhoBERT\_large & SLUE & 0.69 & 0.77 & 0.73 \\ \cline{2-5}  & ViDeBERTa\_base & Mod. SLUE & 0.55 & 0.45 & 0.49 \\ \cline{2-5}  & ViDeBERTa\_base & seqeval & 0.33 & 0.34 & 0.34 \\ \cline{2-5}  & ViDeBERTTa\_base & SLUE & 0.50 & 0.41 & 0.45 \\ \cline{2-5}  & ViT5\_base & Mod. SLUE & 0.65 & 0.76 & 0.70 \\ \cline{2-5}  & ViT5\_base & seqeval & 0.68 & 0.59 & 0.63 \\ \cline{2-5}  & ViT5\_base & SLUE & 0.64 & 0.74 & 0.69 \\ \cline{2-5}  & XLM-R\_base & Mod. SLUE & 0.66 & 0.75 & 0.70 \\ \cline{2-5}  & XLM-R\_base & seqeval & 0.66 & 0.53 & 0.59 \\ \cline{2-5}  & XLM-R\_base & SLUE & 0.64 & 0.73 & 0.69 \\ \cline{2-5}  & XLM-R\_large & Mod. SLUE & 0.72 & 0.78 & 0.75 \\ \cline{2-5}  & XLM-R\_large & seqeval & 0.72 & 0.63 & 0.67 \\ \cline{2-5}  & XLM-R\_large & SLUE & 0.71 & 0.77 & 0.74 \\ \hline \end{tabular}
\end{table}
Table 24: NER results of **Overall Micro** (in percent) on **reference text** of test set using various pre-trained language models. Metrics shown are Precision, Recall, and overall micro/macro F1 score. Evaluation toolkits used are: seqeval, SLUE and our modified SLUE.





\begin{table}
\begin{tabular}{|c|l|l|l|l|l|l|} \hline
**Entity Type** & **NER** & **ASR** & **Eval. Toolkit** & **Prec.** & **Rec.** & **F1** \\ \hline  & ViDeBERTa base & XLSR-53-Viet & SLUE & 0.24 & 0.03 & 0.05 \\ \cline{2-6}  & ViDeBERTa base & XLSR-53-Viet & Mod. SLUE & 0.32 & 0.04 & 0.07 \\ \cline{2-6}  & ViDeBERTa base & w2v2-Viet & SLUE & 0.24 & 0.03 & 0.05 \\ \cline{2-6}  & ViDeBERTa base & w2v2-Viet & Mod. SLUE & 0.33 & 0.04 & 0.07 \\ \cline{2-6}  & ViT5\_base & XLSR-53-Viet & SLUE & 0.63 & 0.49 & 0.55 \\ \cline{2-6}  & ViT5\_base & XLSR-53-Viet & Mod. SLUE & 0.68 & 0.53 & 0.59 \\ \cline{2-6}  & ViT5\_base & w2v2-Viet & SLUE & 0.63 & 0.52 & 0.57 \\ \cline{2-6}  & ViT5\_base & w2v2-Viet & Mod. SLUE & 0.68 & 0.56 & 0.61 \\ \cline{2-6}  & mBART-50 & XLSR-53-Viet & SLUE & 0.44 & 0.06 & 0.11 \\ \cline{2-6}  & mBART-50 & XLSR-53-Viet & Mod. SLUE & 0.55 & 0.08 & 0.14 \\ \cline{2-6}  & mBART-50 & w2v2-Viet & SLUE & 0.44 & 0.08 & 0.13 \\ \cline{2-6}  & mBART-50 & w2v2-Viet & Mod. SLUE & 0.58 & 0.10 & 0.17 \\ \cline{2-6}  & BARTpho & XLSR-53-Viet & SLUE & 0.62 & 0.57 & 0.59 \\ \cline{2-6}  & BARTpho & XLSR-53-Viet & Mod. SLUE & 0.68 & 0.62 & 0.65 \\ \cline{2-6}  & BARTpho & w2v2-Viet & SLUE & 0.59 & 0.58 & 0.58 \\ \cline{2-6}  & BARTpho & w2v2-Viet & Mod. SLUE & 0.64 & 0.63 & 0.63 \\ \cline{2-6}  & PhoBERT\_base-v2 & XLSR-53-Viet & SLUE & 0.72 & 0.58 & 0.64 \\ \cline{2-6}  & PhoBERT\_base-v2 & XLSR-53-Viet & Mod. SLUE & 0.75 & 0.61 & 0.67 \\ \cline{2-6}  & PhoBERT\_base-v2 & w2v2-Viet & SLUE & 0.70 & 0.58 & 0.64 \\ \cline{2-6}  & PhoBERT\_base-v2 & w2v2-Viet & Mod. SLUE & 0.74 & 0.61 & 0.67 \\ \cline{2-6}  & PhoBERT\_base & XLSR-53-Viet & SLUE & 0.69 & 0.60 & 0.64 \\ \cline{2-6}  & WhoBERT\_base & XLSR-53-Viet & Mod. SLUE & 0.74 & 0.64 & 0.68 \\ \cline{2-6}  & PhoBERT\_base & w2v2-Viet & SLUE & 0.67 & 0.61 & 0.64 \\ \cline{2-6}  & PhoBERT\_base & w2v2-Viet & Mod. SLUE & 0.70 & 0.64 & 0.67 \\ \cline{2-6}  & WhoBERT\_large & XLSR-53-Viet & SLUE & 0.71 & 0.59 & 0.65 \\ \cline{2-6}  & WhoBERT\_large & XLSR-53-Viet & Mod. SLUE & 0.75 & 0.62 & 0.68 \\ \cline{2-6}  & PhoBERT\_large & w2v2-Viet & SLUE & 0.69 & 0.60 & 0.64 \\ \cline{2-6}  & WhoBERT\_large & w2v2-Viet & Mod. SLUE & 0.72 & 0.63 & 0.67 \\ \cline{2-6}  & XLM-R\_base & XLSR-53-Viet & SLUE & 0.61 & 0.58 & 0.60 \\ \cline{2-6}  & XLM-R\_base & XLSR-53-Viet & Mod. SLUE & 0.65 & 0.62 & 0.63 \\ \cline{2-6}  & XLM-R\_base & w2v2-Viet & SLUE & 0.59 & 0.59 & 0.59 \\ \cline{2-6}  & XLM-R\_base & w2v2-Viet & Mod. SLUE & 0.63 & 0.63 & 0.63 \\ \cline{2-6}  & XLM-R\_large & XLSR-53-Viet & SLUE & 0.72 & 0.61 & 0.66 \\ \cline{2-6}  & XLM-R\_large & XLSR-53-Viet & Mod. SLUE & 0.76 & 0.64 & 0.69 \\ \cline{2-6}  & XLM-R\_large & w2v2-Viet & SLUE & 0.70 & 0.62 & 0.66 \\ \cline{2-6}  & XLM-R\_large & w2v2-Viet & Mod. SLUE & 0.73 & 0.65 & 0.69 \\ \hline \end{tabular}
\end{table}
Table 25: NER results of **AGE** entity type (in percent) on **ASR output** of test set using various pre-trained language models and ASR models. Metrics shown are Precision, Recall, and overall micro/macro F1 score. Evaluation toolkits used are: SLUE and our modified SLUE.





\begin{table}
\begin{tabular}{|c|l|l|l|l|l|l|} \hline
**Entity Type** & **NER** & **ASR** & **Eval. Toolkit** & **Prec.** & **Rec.** & **F1** \\ \hline  & ViDeBERTa\_base & XLSR-53-Viet & SLUE & 0.52 & 0.56 & 0.54 \\ \cline{2-7}  & ViDeBERTa\_base & XLSR-53-Viet & Mod. SLUE & 0.56 & 0.60 & 0.58 \\ \cline{2-7}  & ViDeBERTa\_base & w2v2-Viet & SLUE & 0.53 & 0.57 & 0.55 \\ \cline{2-7}  & ViDeBERTa\_base & w2v2-Viet & Mod. SLUE & 0.57 & 0.60 & 0.59 \\ \cline{2-7}  & ViT5\_base & XLSR-53-Viet & SLUE & 0.57 & 0.57 & 0.57 \\ \cline{2-7}  & ViT5\_base & XLSR-53-Viet & Mod. SLUE & 0.63 & 0.62 & 0.62 \\ \cline{2-7}  & ViT5\_base & w2v2-Viet & SLUE & 0.60 & 0.56 & 0.58 \\ \cline{2-7}  & ViT5\_base & w2v2-Viet & Mod. SLUE & 0.63 & 0.59 & 0.61 \\ \cline{2-7}  & mBART-50 & XLSR-53-Viet & SLUE & 0.42 & 0.06 & 0.11 \\ \cline{2-7}  & mBART-50 & XLSR-53-Viet & Mod. SLUE & 0.57 & 0.09 & 0.15 \\ \cline{2-7}  & mBART-50 & w2v2-Viet & SLUE & 0.40 & 0.06 & 0.10 \\ \cline{2-7}  & mBART-50 & w2v2-Viet & Mod. SLUE & 0.61 & 0.09 & 0.15 \\ \cline{2-7}  & BARTpho & XLSR-53-Viet & SLUE & 0.64 & 0.62 & 0.63 \\ \cline{2-7}  & BARTpho & XLSR-53-Viet & Mod. SLUE & 0.68 & 0.66 & 0.67 \\ \cline{2-7}  & BARTpho & w2v2-Viet & SLUE & 0.65 & 0.60 & 0.62 \\ \cline{2-7}  & BartPho & w2v2-Viet & Mod. SLUE & 0.69 & 0.64 & 0.66 \\ \cline{2-7}  & PhoBERT\_base-v2 & XLSR-53-Viet & SLUE & 0.67 & 0.69 & 0.68 \\ \cline{2-7} DATETIME & PhoBERT\_base-v2 & XLSR-53-Viet & Mod. SLUE & 0.70 & 0.72 & 0.71 \\ \cline{2-7}  & PhoBERT\_base-v2 & w2v2-Viet & SLUE & 0.69 & 0.69 & 0.69 \\ \cline{2-7}  & PhoBERT\_base-v2 & w2v2-Viet & Mod. SLUE & 0.71 & 0.71 & 0.71 \\ \cline{2-7}  & PhoBERT\_base & XLSR-53-Viet & SLUE & 0.66 & 0.68 & 0.67 \\ \cline{2-7}  & PhoBERT\_base & XLSR-53-Viet & Mod. SLUE & 0.69 & 0.72 & 0.70 \\ \cline{2-7}  & PhoBERT\_base & w2v2-Viet & SLUE & 0.68 & 0.67 & 0.68 \\ \cline{2-7}  & WhoBERT\_base & w2v2-Viet & Mod. SLUE & 0.71 & 0.70 & 0.70 \\ \cline{2-7}  & PhoBERT\_large & XLSR-53-Viet & SLUE & 0.67 & 0.68 & 0.68 \\ \cline{2-7}  & ThobBERT\_large & XLSR-53-Viet & Mod. SLUE & 0.70 & 0.71 & 0.71 \\ \cline{2-7}  & PhoBERT\_large & w2v2-Viet & SLUE & 0.70 & 0.66 & 0.68 \\ \cline{2-7}  & WhoBERT\_large & w2v2-Viet & Mod. SLUE & 0.72 & 0.69 & 0.70 \\ \cline{2-7}  & XLM-R\_base & XLSR-53-Viet & SLUE & 0.64 & 0.65 & 0.64 \\ \cline{2-7}  & XLM-R\_base & XLSR-53-Viet & Mod. SLUE & 0.67 & 0.68 & 0.68 \\ \cline{2-7}  & XLM-R\_base & w2v2-Viet & SLUE & 0.66 & 0.64 & 0.65 \\ \cline{2-7}  & XLM-R\_base & w2v2-Viet & Mod. SLUE & 0.69 & 0.66 & 0.67 \\ \cline{2-7}  & XLM-R\_large & XLSR-53-Viet & SLUE & 0.69 & 0.67 & 0.68 \\ \cline{2-7}  & XLM-R\_large & XLSR-53-Viet & Mod. SLUE & 0.72 & 0.70 & 0.71 \\ \cline{2-7}  & XLM-R\_large & w2v2-Viet & SLUE & 0.72 & 0.66 & 0.69 \\ \cline{2-7}  & XLM-R\_large & w2v2-Viet & Mod. SLUE & 0.74 & 0.69 & 0.71 \\ \hline \end{tabular}
\end{table}
Table 26: NER results of **DATETIME** entity type (in percent) on **ASR output** of test set using various pre-trained language models and ASR models. Metrics shown are Precision, Recall, and overall micro/macro F1 score. Evaluation toolkits used are: SLUE and our modified SLUE.





\begin{table}
\begin{tabular}{|c|l|l|l|l|l|l|} \hline
**Entity Type** & **NER** & **ASR** & **Eval. Toolkit** & **Prec.** & **Rec.** & **F1** \\ \hline  & ViDeBERTa\_base & XLSR-53-Viet & SLUE & 0.53 & 0.47 & 0.50 \\ \cline{2-6}  & ViDeBERTa\_base & XLSR-53-Viet & Mod. SLUE & 0.55 & 0.49 & 0.52 \\ \cline{2-6}  & ViDeBERTa\_base & w2v2-Viet & SLUE & 0.51 & 0.44 & 0.47 \\ \cline{2-6}  & ViDeBERTa\_base & w2v2-Viet & Mod. SLUE & 0.53 & 0.46 & 0.49 \\ \cline{2-6}  & ViT5\_base & XLSR-53-Viet & SLUE & 0.52 & 0.47 & 0.50 \\ \cline{2-6}  & ViT5\_base & XLSR-53-Viet & Mod. SLUE & 0.54 & 0.49 & 0.51 \\ \cline{2-6}  & ViT5\_base & w2v2-Viet & SLUE & 0.53 & 0.46 & 0.49 \\ \cline{2-6}  & ViT5\_base & w2v2-Viet & Mod. SLUE & 0.53 & 0.46 & 0.49 \\ \cline{2-6}  & miBART-50 & XLSR-53-Viet & SLUE & 0.30 & 0.08 & 0.13 \\ \cline{2-6}  & mBART-50 & XLSR-53-Viet & Mod. SLUE & 0.34 & 0.10 & 0.15 \\ \cline{2-6}  & mBART-50 & w2v2-Viet & SLUE & 0.41 & 0.06 & 0.11 \\ \cline{2-6}  & mBART-50 & w2v2-Viet & Mod. SLUE & 0.48 & 0.08 & 0.13 \\ \cline{2-6}  & BARTrho & XLSR-53-Viet & SLUE & 0.59 & 0.56 & 0.58 \\ \cline{2-6}  & BARTrho & XLSR-53-Viet & Mod. SLUE & 0.62 & 0.59 & 0.60 \\ \cline{2-6}  & BARTrho & w2v2-Viet & SLUE & 0.60 & 0.53 & 0.56 \\ \cline{2-6}  & BARTrho & w2v2-Viet & Mod. SLUE & 0.63 & 0.56 & 0.59 \\ \cline{2-6}  & PhoBERT\_base-v2 & XLSR-53-Viet & SLUE & 0.56 & 0.57 & 0.57 \\ \cline{2-6}  & PhoBERT\_base-v2 & XLSR-53-Viet & Mod. SLUE & 0.58 & 0.59 & 0.58 \\ \cline{2-6} DIAGNOSTICS & PhoBERT\_base-v2 & WL2v2-Viet & SLUE & 0.56 & 0.56 & 0.56 \\ \cline{2-6}  & PhoBERT\_base-v2 & w2v2-Viet & Mod. SLUE & 0.59 & 0.59 & 0.59 \\ \cline{2-6}  & PhoBERT\_base & XLSR-53-Viet & SLUE & 0.57 & 0.57 & 0.57 \\ \cline{2-6}  & PhoBERT\_base & XLSR-53-Viet & Mod. SLUE & 0.59 & 0.59 & 0.59 \\ \cline{2-6}  & PhoBERT\_base & w2v2-Viet & SLUE & 0.55 & 0.56 & 0.55 \\ \cline{2-6}  & PhoBERT\_base & w2v2-Viet & Mod. SLUE & 0.57 & 0.58 & 0.57 \\ \cline{2-6}  & WhoBERT\_large & XLSR-53-Viet & SLUE & 0.61 & 0.58 & 0.59 \\ \cline{2-6}  & ThoBERT\_large & XLSR-53-Viet & Mod. SLUE & 0.62 & 0.59 & 0.61 \\ \cline{2-6}  & PhoBERT\_large & w2v2-Viet & SLUE & 0.58 & 0.57 & 0.57 \\ \cline{2-6}  & WhoBERT\_large & w2v2-Viet & Mod. SLUE & 0.61 & 0.59 & 0.60 \\ \cline{2-6}  & XLM-R\_base & XLSR-53-Viet & SLUE & 0.51 & 0.58 & 0.54 \\ \cline{2-6}  & XLM-R\_base & XLSR-53-Viet & Mod. SLUE & 0.52 & 0.59 & 0.55 \\ \cline{2-6}  & XLM-R\_base & w2v2-Viet & SLUE & 0.50 & 0.57 & 0.53 \\ \cline{2-6}  & XM-R\_base & w2v2-Viet & Mod. SLUE & 0.52 & 0.59 & 0.55 \\ \cline{2-6}  & XLM-R\_large & XLSR-53-Viet & SLUE & 0.59 & 0.58 & 0.59 \\ \cline{2-6}  & XLM-R\_large & XLSR-53-Viet & Mod. SLUE & 0.61 & 0.59 & 0.60 \\ \cline{2-6}  & XLM-R\_large & w2v2-Viet & SLUE & 0.58 & 0.57 & 0.57 \\ \cline{2-6}  & XMM-R\_large & w2v2-Viet & Mod. SLUE & 0.60 & 0.59 & 0.60 \\ \hline \end{tabular}
\end{table}
Table 27: NER results of **DIAGNOSTICS** entity type (in percent) on **ASR output** of test set using various pre-trained language models and ASR models. Metrics shown are Precision, Recall, and overall micro/macro F1 score. Evaluation toolkits used are: SLUE and our modified SLUE.





\begin{table}
\begin{tabular}{|c|l|l|l|r|r|r|} \hline
**Entity Type** & **NER** & **ASR** & **Eval. Toolkit** & **Prec.** & **Rec.** & **F1** \\ \hline  & ViDeBERTa\_base & XLSR-53-Viet & SLUE & 0.53 & 0.39 & 0.45 \\ \cline{2-6}  & ViDeBERTa\_base & XLSR-53-Viet & Mod. SLUE & 0.65 & 0.48 & 0.55 \\ \cline{2-6}  & ViDeBERTa\_base & w2v2-Viet & SLUE & 0.53 & 0.39 & 0.45 \\ \cline{2-6}  & ViDeBERTa\_base & w2v2-Viet & Mod. SLUE & 0.64 & 0.48 & 0.55 \\ \cline{2-6}  & ViT5\_base & XLSR-53-Viet & SLUE & 0.59 & 0.49 & 0.54 \\ \cline{2-6}  & ViT5\_base & XLSR-53-Viet & Mod. SLUE & 0.69 & 0.58 & 0.63 \\ \cline{2-6}  & ViT5\_base & w2v2-Viet & SLUE & 0.59 & 0.49 & 0.54 \\ \cline{2-6}  & ViT5\_base & w2v2-Viet & Mod. SLUE & 0.70 & 0.58 & 0.64 \\ \cline{2-6}  & mBART-50 & XLSR-53-Viet & SLUE & 0.51 & 0.09 & 0.15 \\ \cline{2-6}  & mBART-50 & XLSR-53-Viet & Mod. SLUE & 0.64 & 0.11 & 0.19 \\ \cline{2-6}  & mBART-50 & w2v2-Viet & SLUE & 0.48 & 0.07 & 0.12 \\ \cline{2-6}  & mBART-50 & w2v2-Viet & Mod. SLUE & 0.65 & 0.09 & 0.16 \\ \cline{2-6}  & BARTPho & XLSR-53-Viet & SLUE & 0.61 & 0.53 & 0.57 \\ \cline{2-6}  & BARTpho & XLSR-53-Viet & Mod. SLUE & 0.71 & 0.62 & 0.66 \\ \cline{2-6}  & BARTPho & w2v2-Viet & SLUE & 0.58 & 0.53 & 0.55 \\ \cline{2-6}  & BARTPho & w2v2-Viet & Mod. SLUE & 0.69 & 0.62 & 0.65 \\ \cline{2-6}  & PhoBERT\_base-v2 & XLSR-53-Viet & SLUE & 0.63 & 0.57 & 0.60 \\ \cline{2-6}  & PhoBERT\_base-v2 & XLSR-53-Viet & Mod. SLUE & 0.71 & 0.65 & 0.68 \\ \cline{2-6}  & PhoBERT\_base-v2 & w2v2-Viet & SLUE & 0.61 & 0.57 & 0.59 \\ \cline{2-6}  & PhoBERT\_base-v2 & w2v2-Viet & Mod. SLUE & 0.71 & 0.66 & 0.68 \\ \cline{2-6}  & PhoBERT\_base-v2 & XLSR-53-Viet & SLUE & 0.62 & 0.56 & 0.59 \\ \cline{2-6}  & PhoBERT\_base & XLSR-53-Viet & Mod. SLUE & 0.71 & 0.64 & 0.67 \\ \cline{2-6}  & WhoBERT\_base & w2v2-Viet & SLUE & 0.61 & 0.55 & 0.58 \\ \cline{2-6}  & PhoBERT\_base & w2v2-Viet & Mod. SLUE & 0.71 & 0.64 & 0.67 \\ \cline{2-6}  & WhoBERT\_large & XLSR-53-Viet & SLUE & 0.63 & 0.56 & 0.59 \\ \cline{2-6}  & ThoBERT\_large & XLSR-53-Viet & Mod. SLUE & 0.71 & 0.63 & 0.67 \\ \cline{2-6}  & PhoBERT\_large & w2v2-Viet & SLUE & 0.62 & 0.54 & 0.58 \\ \cline{2-6}  & WhoBERT\_large & w2v2-Viet & Mod. SLUE & 0.71 & 0.62 & 0.66 \\ \cline{2-6}  & XLM-R\_base & XLSR-53-Viet & SLUE & 0.58 & 0.55 & 0.57 \\ \cline{2-6}  & XLM-R\_base & XLSR-53-Viet & Mod. SLUE & 0.67 & 0.64 & 0.65 \\ \cline{2-6}  & XLM-R\_base & w2v2-Viet & SLUE & 0.57 & 0.54 & 0.56 \\ \cline{2-6}  & XM-R\_base & w2v2-Viet & Mod. SLUE & 0.67 & 0.64 & 0.65 \\ \cline{2-6}  & XLM-R\_large & XLSR-53-Viet & SLUE & 0.64 & 0.57 & 0.60 \\ \cline{2-6}  & XLM-R\_large & XLSR-53-Viet & Mod. SLUE & 0.72 & 0.64 & 0.68 \\ \cline{2-6}  & XLM-R\_large & w2v2-Viet & SLUE & 0.64 & 0.56 & 0.60 \\ \cline{2-6}  & XLM-R\_large & w2v2-Viet & Mod. SLUE & 0.72 & 0.63 & 0.67 \\ \hline \end{tabular}
\end{table}
Table 28: NER results of **DISEASSYMTOM** entity type (in percent) on **ASR output** of test set using various pre-trained language models and ASR models. Metrics shown are Precision, Recall, and overall micro/macro F1 score. Evaluation toolkits used are: SLUE and our modified SLUE.





\begin{table}
\begin{tabular}{|c|l|l|l|l|l|l|} \hline
**Entity Type** & **NER** & **ASR** & **Eval. Toolkit** & **Prec.** & **Rec.** & **F1** \\ \hline  & ViDeBERTa\_base & XLSR-53-Viet & SLUE & 0.51 & 0.34 & 0.41 \\ \cline{2-6}  & ViDeBERTa\_base & XLSR-53-Viet & Mod. SLUE & 0.59 & 0.40 & 0.47 \\ \cline{2-6}  & ViDeBERTa\_base & w2v2-Viet & SLUE & 0.49 & 0.34 & 0.40 \\ \cline{2-6}  & ViDeBERTa\_base & w2v2-Viet & Mod. SLUE & 0.56 & 0.39 & 0.46 \\ \cline{2-6}  & ViT5\_base & XLSR-53-Viet & SLUE & 0.58 & 0.52 & 0.55 \\ \cline{2-6}  & ViT5\_base & XLSR-53-Viet & Mod. SLUE & 0.66 & 0.59 & 0.62 \\ \cline{2-6}  & ViT5\_base & w2v2-Viet & SLUE & 0.60 & 0.51 & 0.55 \\ \cline{2-6}  & ViT5\_base & w2v2-Viet & Mod. SLUE & 0.68 & 0.58 & 0.62 \\ \cline{2-6}  & mBART-50 & XLSR-53-Viet & SLUE & 0.32 & 0.06 & 0.11 \\ \cline{2-6}  & mBART-50 & XLSR-53-Viet & Mod. SLUE & 0.46 & 0.09 & 0.15 \\ \cline{2-6}  & mBART-50 & w2v2-Viet & SLUE & 0.35 & 0.07 & 0.11 \\ \cline{2-6}  & mBART-50 & w2v2-Viet & Mod. SLUE & 0.54 & 0.10 & 0.17 \\ \cline{2-6}  & BARTPho & XLSR-53-Viet & SLUE & 0.60 & 0.55 & 0.57 \\ \cline{2-6}  & BARTpho & XLSR-53-Viet & Mod. SLUE & 0.67 & 0.62 & 0.64 \\ \cline{2-6}  & BARTPho & w2v2-Viet & SLUE & 0.60 & 0.55 & 0.57 \\ \cline{2-6}  & BARTPho & w2v2-Viet & Mod. SLUE & 0.66 & 0.61 & 0.64 \\ \cline{2-6}  & PhoBERT\_base-v2 & XLSR-53-Viet & SLUE & 0.67 & 0.66 & 0.67 \\ \cline{2-6} DRUGCHEMICAL & PhoBERT\_base-v2 & XLSR-53-Viet & Mod. SLUE & 0.73 & 0.72 & 0.72 \\ \cline{2-6}  & PhoBERT\_base-v2 & w2v2-Viet & SLUE & 0.66 & 0.66 & 0.66 \\ \cline{2-6}  & PhoBERT\_base-v2 & w2v2-Viet & Mod. SLUE & 0.72 & 0.72 & 0.72 \\ \cline{2-6}  & PhoBERT\_base & XLSR-53-Viet & SLUE & 0.66 & 0.66 & 0.66 \\ \cline{2-6}  & PhoBERT\_base & XLSR-53-Viet & Mod. SLUE & 0.72 & 0.72 & 0.72 \\ \cline{2-6}  & PhoBERT\_base & w2v2-Viet & SLUE & 0.64 & 0.66 & 0.65 \\ \cline{2-6}  & WhoBERT\_base & w2v2-Viet & SLUE & 0.63 & 0.67 & 0.65 \\ \cline{2-6}  & ThoBERT\_large & XLSR-53-Viet & Mod. SLUE & 0.69 & 0.72 & 0.71 \\ \cline{2-6}  & PhoBERT\_large & w2v2-Viet & SLUE & 0.64 & 0.66 & 0.65 \\ \cline{2-6}  & XLM-R\_base & XLSR-53-Viet & SLUE & 0.63 & 0.52 & 0.57 \\ \cline{2-6}  & XLM-R\_base & XLSR-53-Viet & Mod. SLUE & 0.71 & 0.57 & 0.63 \\ \cline{2-6}  & XMM-R\_base & w2v2-Viet & SLUE & 0.64 & 0.52 & 0.57 \\ \cline{2-6}  & XMM-R\_base & w2v2-Viet & Mod. SLUE & 0.71 & 0.57 & 0.64 \\ \cline{2-6}  & XMM-R\_large & XLSR-53-Viet & SLUE & 0.70 & 0.68 & 0.69 \\ \cline{2-6}  & XLM-R\_large & XLSR-53-Viet & Mod. SLUE & 0.76 & 0.73 & 0.74 \\ \cline{2-6}  & XMM-R\_large & w2v2-Viet & SLUE & 0.69 & 0.66 & 0.67 \\ \cline{2-6}  & XMM-R\_large & w2v2-Viet & Mod. SLUE & 0.75 & 0.71 & 0.73 \\ \hline \end{tabular}
\end{table}
Table 29: NER results of **DRUGCHEMICAL** entity type (in percent) on **ASR output** of test set using various pre-trained language models and ASR models. Metrics shown are Precision, Recall, and overall micro/macro F1 score. Evaluation toolkits used are: SLUE and our modified SLUE.





\begin{table}
\begin{tabular}{|c|l|l|l|l|l|l|} \hline
**Entity Type** & **NER** & **ASR** & **Eval. Toolkit** & **Prec.** & **Rec.** & **F1** \\ \hline  & ViDeBERTa\_base & XLSR-53-Viet & SLUE & 0.25 & 0.29 & 0.27 \\ \cline{2-6}  & ViDeBERTa\_base & XLSR-53-Viet & Mod. SLUE & 0.28 & 0.33 & 0.30 \\ \cline{2-6}  & ViDeBERTa\_base & w2v2-Viet & SLUE & 0.22 & 0.27 & 0.24 \\ \cline{2-6}  & ViDeBERTa\_base & w2v2-Viet & Mod. SLUE & 0.25 & 0.31 & 0.28 \\ \cline{2-6}  & ViT5\_base & XLSR-53-Viet & SLUE & 0.54 & 0.53 & 0.54 \\ \cline{2-6}  & ViT5\_base & XLSR-53-Viet & Mod. SLUE & 0.58 & 0.56 & 0.57 \\ \cline{2-6}  & ViT5\_base & w2v2-Viet & SLUE & 0.55 & 0.53 & 0.54 \\ \cline{2-6}  & ViT5\_base & w2v2-Viet & Mod. SLUE & 0.59 & 0.58 & 0.59 \\ \cline{2-6}  & mBART-50 & XLSR-53-Viet & SLUE & 0.59 & 0.06 & 0.12 \\ \cline{2-6}  & mBART-50 & XLSR-53-Viet & Mod. SLUE & 0.61 & 0.07 & 0.12 \\ \cline{2-6}  & mBART-50 & w2v2-Viet & SLUE & 0.54 & 0.05 & 0.09 \\ \cline{2-6}  & mBART-50 & w2v2-Viet & Mod. SLUE & 0.63 & 0.06 & 0.11 \\ \cline{2-6}  & BARTPho & XLSR-53-Viet & SLUE & 0.65 & 0.49 & 0.56 \\ \cline{2-6}  & BARTpho & XLSR-53-Viet & Mod. SLUE & 0.68 & 0.51 & 0.58 \\ \cline{2-6}  & BARTpho & w2v2-Viet & SLUE & 0.68 & 0.52 & 0.59 \\ \cline{2-6}  & BARTpho & w2v2-Viet & Mod. SLUE & 0.71 & 0.54 & 0.61 \\ \cline{2-6}  & PhoBERT\_base-v2 & XLSR-53-Viet & SLUE & 0.68 & 0.68 & 0.68 \\ \cline{2-6}  & PhoBERT\_base-v2 & XLSR-53-Viet & Mod. SLUE & 0.69 & 0.68 & 0.68 \\ \cline{2-6}  & PhoBERT\_base-v2 & w2v2-Viet & SLUE & 0.69 & 0.66 & 0.67 \\ \cline{2-6}  & PhoBERT\_base-v2 & w2v2-Viet & Mod. SLUE & 0.69 & 0.67 & 0.68 \\ \cline{2-6}  & PhoBERT\_base & XLSR-53-Viet & SLUE & 0.63 & 0.65 & 0.64 \\ \cline{2-6}  & WhoBERT\_base & XLSR-53-Viet & Mod. SLUE & 0.64 & 0.66 & 0.65 \\ \cline{2-6}  & PhoBERT\_base & w2v2-Viet & SLUE & 0.64 & 0.62 & 0.63 \\ \cline{2-6}  & PhoBERT\_base & w2v2-Viet & Mod. SLUE & 0.66 & 0.64 & 0.65 \\ \cline{2-6}  & PhoBERT\_large & XLSR-53-Viet & SLUE & 0.68 & 0.69 & 0.68 \\ \cline{2-6}  & ChoBERT\_large & XLSR-53-Viet & Mod. SLUE & 0.69 & 0.70 & 0.69 \\ \cline{2-6}  & PhoBERT\_large & w2v2-Viet & SLUE & 0.67 & 0.67 & 0.67 \\ \cline{2-6}  & PhoBERT\_large & w2v2-Viet & Mod. SLUE & 0.68 & 0.68 & 0.68 \\ \cline{2-6}  & XLM-R\_base & XLSR-53-Viet & SLUE & 0.52 & 0.63 & 0.57 \\ \cline{2-6}  & XLM-R\_base & XLSR-53-Viet & Mod. SLUE & 0.55 & 0.67 & 0.61 \\ \cline{2-6}  & XLM-R\_base & w2v2-Viet & SLUE & 0.53 & 0.63 & 0.57 \\ \cline{2-6}  & XMM-R\_base & w2v2-Viet & Mod. SLUE & 0.56 & 0.67 & 0.61 \\ \cline{2-6}  & XLM-R\_large & XLSR-53-Viet & SLUE & 0.75 & 0.69 & 0.72 \\ \cline{2-6}  & XLM-R\_large & XLSR-53-Viet & Mod. SLUE & 0.76 & 0.70 & 0.73 \\ \cline{2-6}  & XMM-R\_large & w2v2-Viet & SLUE & 0.76 & 0.68 & 0.72 \\ \cline{2-6}  & XMM-R\_large & w2v2-Viet & Mod. SLUE & 0.77 & 0.69 & 0.73 \\ \hline \end{tabular}
\end{table}
Table 30: NER results of **FOODDRINK** entity type (in percent) on **ASR output** of test set using various pre-trained language models and ASR models. Metrics shown are Precision, Recall, and overall micro/macro F1 score. Evaluation toolkits used are: SLUE and our modified SLUE.





\begin{table}
\begin{tabular}{|c|l|l|l|l|l|l|} \hline
**Entity Type** & **NER** & **ASR** & **Eval. Toolkit** & **Prec.** & **Rec.** & **F1** \\ \hline  & ViDeBERTa base & XLSR-53-Viet & SLUE & 0.00 & 0.00 & 0.00 \\ \cline{2-6}  & ViDeBERTa base & XLSR-53-Viet & Mod. SLUE & 0.00 & 0.00 & 0.00 \\ \cline{2-6}  & ViDeBERTa base & w2v2-Viet & SLUE & 1,00 & 0.00 & 0.00 \\ \cline{2-6}  & ViDeBERTa base & w2v2-Viet & Mod. SLUE & 1,00 & 0.00 & 0.00 \\ \cline{2-6}  & ViT5\_base & XLSR-53-Viet & SLUE & 0.74 & 0.51 & 0.60 \\ \cline{2-6}  & ViT5\_base & XLSR-53-Viet & Mod. SLUE & 0.79 & 0.54 & 0.65 \\ \cline{2-6}  & ViT5\_base & w2v2-Viet & SLUE & 0.74 & 0.51 & 0.60 \\ \cline{2-6}  & ViT5\_base & w2v2-Viet & Mod. SLUE & 0.76 & 0.53 & 0.62 \\ \cline{2-6}  & mBART-50 & XLSR-53-Viet & SLUE & 0.39 & 0.03 & 0.06 \\ \cline{2-6}  & mBART-50 & XLSR-53-Viet & Mod. SLUE & 0.50 & 0.04 & 0.08 \\ \cline{2-6}  & mBART-50 & w2v2-Viet & SLUE & 0.40 & 0.05 & 0.09 \\ \cline{2-6}  & mBART-50 & w2v2-Viet & Mod. SLUE & 0.53 & 0.07 & 0.12 \\ \cline{2-6}  & BARTpho & XLSR-53-Viet & SLUE & 0.78 & 0.59 & 0.67 \\ \cline{2-6}  & BARTpho & XLSR-53-Viet & Mod. SLUE & 0.82 & 0.61 & 0.70 \\ \cline{2-6}  & BARTpho & w2v2-Viet & SLUE & 0.78 & 0.58 & 0.66 \\ \cline{2-6}  & BARTpho & w2v2-Viet & Mod. SLUE & 0.80 & 0.59 & 0.68 \\ \cline{2-6}  & PhoBERT\_base-v2 & XLSR-53-Viet & SLUE & 0.78 & 0.60 & 0.68 \\ \cline{2-6} GENDER & PhoBERT\_base-v2 & XLSR-53-Viet & Mod. SLUE & 0.82 & 0.63 & 0.71 \\ \cline{2-6}  & PhoBERT\_base-v2 & w2v2-Viet & SLUE & 0.78 & 0.60 & 0.68 \\ \cline{2-6}  & PhoBERT\_base-v2 & w2v2-Viet & Mod. SLUE & 0.80 & 0.62 & 0.70 \\ \cline{2-6}  & PhoBERT\_base & XLSR-53-Viet & SLUE & 0.78 & 0.61 & 0.68 \\ \cline{2-6}  & WhoBERT\_base & XLSR-53-Viet & Mod. SLUE & 0.81 & 0.63 & 0.71 \\ \cline{2-6}  & PhoBERT\_base & w2v2-Viet & SLUE & 0.77 & 0.61 & 0.68 \\ \cline{2-6}  & PhoBERT\_base & w2v2-Viet & Mod. SLUE & 0.80 & 0.62 & 0.70 \\ \cline{2-6}  & WhoBERT\_large & XLSR-53-Viet & SLUE & 0.75 & 0.60 & 0.67 \\ \cline{2-6}  & XLSR-53-Viet & Mod. SLUE & 0.78 & 0.63 & 0.70 \\ \cline{2-6}  & PhoBERT\_large & w2v2-Viet & SLUE & 0.76 & 0.60 & 0.67 \\ \cline{2-6}  & WhoBERT\_large & w2v2-Viet & Mod. SLUE & 0.78 & 0.62 & 0.69 \\ \cline{2-6}  & XLM-R\_base & XLSR-53-Viet & SLUE & 0.72 & 0.42 & 0.53 \\ \cline{2-6}  & XLM-R\_base & XLSR-53-Viet & Mod. SLUE & 0.78 & 0.45 & 0.57 \\ \cline{2-6}  & XLM-R\_base & w2v2-Viet & SLUE & 0.70 & 0.44 & 0.54 \\ \cline{2-6}  & XLM-R\_base & w2v2-Viet & Mod. SLUE & 0.74 & 0.47 & 0.57 \\ \cline{2-6}  & XLM-R\_large & XLSR-53-Viet & SLUE & 0.79 & 0.63 & 0.70 \\ \cline{2-6}  & XLM-R\_large & XLSR-53-Viet & Mod. SLUE & 0.82 & 0.66 & 0.73 \\ \cline{2-6}  & XLM-R\_large & w2v2-Viet & SLUE & 0.79 & 0.64 & 0.71 \\ \cline{2-6}  & XLM-R\_large & w2v2-Viet & Mod. SLUE & 0.81 & 0.65 & 0.72 \\ \hline \end{tabular}
\end{table}
Table 31: NER results of **GENDER** entity type (in percent) on **ASR output** of test set using various pre-trained language models and ASR models. Metrics shown are Precision, Recall, and overall micro/macro F1 score. Evaluation toolkits used are: SLUE and our modified SLUE.





\begin{table}
\begin{tabular}{|c|l|l|l|l|l|l|} \hline
**Entity Type** & **NER** & **ASR** & **Eval. Toolkit** & **Prec.** & **Rec.** & **F1** \\ \hline  & ViDeBERTa\_base & XLSR-53-Viet & SLUE & 0.63 & 0.28 & 0.39 \\ \cline{2-7}  & ViDeBERTa\_base & XLSR-53-Viet & Mod. SLUE & 0.67 & 0.29 & 0.41 \\ \cline{2-7}  & ViDeBERTa\_base & w2v2-Viet & SLUE & 0.59 & 0.26 & 0.36 \\ \cline{2-7}  & ViDeBERTa\_base & w2v2-Viet & Mod. SLUE & 0.60 & 0.26 & 0.37 \\ \cline{2-7}  & ViT5\_base & XLSR-53-Viet & SLUE & 0.62 & 0.57 & 0.59 \\ \cline{2-7}  & ViT5\_base & XLSR-53-Viet & Mod. SLUE & 0.65 & 0.60 & 0.63 \\ \cline{2-7}  & ViT5\_base & w2v2-Viet & SLUE & 0.61 & 0.53 & 0.57 \\ \cline{2-7}  & ViT5\_base & w2v2-Viet & Mod. SLUE & 0.64 & 0.56 & 0.60 \\ \cline{2-7}  & mBART-50 & XLSR-53-Viet & SLUE & 0.18 & 0.01 & 0.02 \\ \cline{2-7}  & mBART-50 & XLSR-53-Viet & Mod. SLUE & 0.41 & 0.02 & 0.04 \\ \cline{2-7}  & mBART-50 & w2v2-Viet & SLUE & 0.26 & 0.01 & 0.03 \\ \cline{2-7}  & mBART-50 & w2v2-Viet & Mod. SLUE & 0.33 & 0.02 & 0.04 \\ \cline{2-7}  & BARTpho & XLSR-53-Viet & SLUE & 0.65 & 0.66 & 0.65 \\ \cline{2-7}  & BARTpho & XLSR-53-Viet & Mod. SLUE & 0.68 & 0.69 & 0.69 \\ \cline{2-7}  & BARTpho & w2v2-Viet & SLUE & 0.65 & 0.65 & 0.65 \\ \cline{2-7}  & BARTpho & w2v2-Viet & Mod. SLUE & 0.69 & 0.68 & 0.68 \\ \cline{2-7}  & PhoBERT\_base-v2 & XLSR-53-Viet & SLUE & 0.66 & 0.71 & 0.69 \\ \cline{2-7}  & PhoBERT\_base-v2 & XLSR-53-Viet & Mod. SLUE & 0.69 & 0.74 & 0.71 \\ \cline{2-7}  & PhoBERT\_base-v2 & w2v2-Viet & SLUE & 0.67 & 0.69 & 0.68 \\ \cline{2-7}  & PhoBERT\_base-v2 & w2v2-Viet & Mod. SLUE & 0.70 & 0.72 & 0.71 \\ \cline{2-7}  & PhoBERT\_base & XLSR-53-Viet & SLUE & 0.62 & 0.68 & 0.65 \\ \cline{2-7}  & PhoBERT\_base & XLSR-53-Viet & Mod. SLUE & 0.66 & 0.72 & 0.69 \\ \cline{2-7}  & WhoBERT\_base & w2v2-Viet & SLUE & 0.64 & 0.67 & 0.65 \\ \cline{2-7}  & PhoBERT\_base & w2v2-Viet & Mod. SLUE & 0.67 & 0.71 & 0.69 \\ \cline{2-7}  & WhoBERT\_large & XLSR-53-Viet & SLUE & 0.67 & 0.68 & 0.67 \\ \cline{2-7}  & WhoBERT\_large & XLSR-53-Viet & Mod. SLUE & 0.69 & 0.70 & 0.70 \\ \cline{2-7}  & PhoBERT\_large & w2v2-Viet & SLUE & 0.70 & 0.66 & 0.68 \\ \cline{2-7}  & WhoBERT\_large & w2v2-Viet & Mod. SLUE & 0.72 & 0.68 & 0.70 \\ \cline{2-7}  & XLM-R\_base & XLSR-53-Viet & SLUE & 0.64 & 0.61 & 0.62 \\ \cline{2-7}  & XLM-R\_base & XLSR-53-Viet & Mod. SLUE & 0.67 & 0.64 & 0.65 \\ \cline{2-7}  & XLM-R\_base & w2v2-Viet & SLUE & 0.65 & 0.59 & 0.62 \\ \cline{2-7}  & XLM-R\_base & w2v2-Viet & Mod. SLUE & 0.68 & 0.62 & 0.65 \\ \cline{2-7}  & XLM-R\_large & XLSR-53-Viet & SLUE & 0.71 & 0.69 & 0.70 \\ \cline{2-7}  & XLM-R\_large & XLSR-53-Viet & Mod. SLUE & 0.74 & 0.72 & 0.73 \\ \cline{2-7}  & XLM-R\_large & w2v2-Viet & SLUE & 0.72 & 0.67 & 0.69 \\ \cline{2-7}  & XLM-R\_large & w2v2-Viet & Mod. SLUE & 0.74 & 0.70 & 0.72 \\ \hline \end{tabular}
\end{table}
Table 32: NER results of **LOCATION** entity type (in percent) on **ASR output** of test set using various pre-trained language models and ASR models. Metrics shown are Precision, Recall, and overall micro/macro F1 score. Evaluation toolkits used are: SLUE and our modified SLUE.





\begin{table}
\begin{tabular}{|c|l|c|l|l|l|l|} \hline
**Entity Type** & **NER** & **ASR** & **Eval. Toolkit** & **Prec.** & **Rec.** & **F1** \\ \hline  & ViDeBERTa\_base & XLSR-53-Viet & SLUE & 0.22 & 0.07 & 0.10 \\ \cline{2-6}  & ViDeBERTa\_base & XLSR-53-Viet & Mod. SLUE & 0.33 & 0.10 & 0.16 \\ \cline{2-6}  & ViDeBERTa\_base & w2v2-Viet & SLUE & 0.20 & 0.06 & 0.10 \\ \cline{2-6}  & ViDeBERTa\_base & w2v2-Viet & Mod. SLUE & 0.33 & 0.11 & 0.16 \\ \cline{2-6}  & ViT5\_base & XLSR-53-Viet & SLUE & 0.39 & 0.09 & 0.14 \\ \cline{2-6}  & ViT5\_base & XLSR-53-Viet & Mod. SLUE & 0.48 & 0.11 & 0.18 \\ \cline{2-6}  & ViT5\_base & w2v2-Viet & SLUE & 0.44 & 0.11 & 0.17 \\ \cline{2-6}  & ViT5\_base & w2v2-Viet & Mod. SLUE & 0.53 & 0.13 & 0.21 \\ \cline{2-6}  & mBART-50 & XLSR-53-Viet & SLUE & 0.31 & 0.01 & 0.01 \\ \cline{2-6}  & mBART-50 & XLSR-53-Viet & Mod. SLUE & 0.38 & 0.01 & 0.01 \\ \cline{2-6}  & mBART-50 & w2v2-Viet & SLUE & 0.26 & 0.01 & 0.01 \\ \cline{2-6}  & mBART-50 & w2v2-Viet & Mod. SLUE & 0.32 & 0.01 & 0.01 \\ \cline{2-6}  & BARTPho & XLSR-53-Viet & SLUE & 0.42 & 0.09 & 0.15 \\ \cline{2-6}  & BARTPho & XLSR-53-Viet & Mod. SLUE & 0.51 & 0.11 & 0.18 \\ \cline{2-6}  & BARTPho & w2v2-Viet & SLUE & 0.38 & 0.09 & 0.14 \\ \cline{2-6}  & BaRTpho & w2v2-Viet & Mod. SLUE & 0.49 & 0.11 & 0.18 \\ \cline{2-6}  & PhoBERT\_base-v2 & XLSR-53-Viet & SLUE & 0.46 & 0.20 & 0.28 \\ \cline{2-6}  & PhoBERT\_base-v2 & XLSR-53-Viet & Mod. SLUE & 0.56 & 0.24 & 0.33 \\ \cline{2-6}  & PhoBERT\_base-v2 & w2v2-Viet & SLUE & 0.43 & 0.18 & 0.25 \\ \cline{2-6}  & PhoBERT\_base-v2 & w2v2-Viet & Mod. SLUE & 0.52 & 0.22 & 0.30 \\ \cline{2-6}  & PhoBERT\_base & XLSR-53-Viet & SLUE & 0.48 & 0.22 & 0.30 \\ \cline{2-6}  & WhoBERT\_base & XLSR-53-Viet & Mod. SLUE & 0.56 & 0.26 & 0.35 \\ \cline{2-6}  & PhoBERT\_base & w2v2-Viet & SLUE & 0.44 & 0.20 & 0.28 \\ \cline{2-6}  & PhoBERT\_base & w2v2-Viet & Mod. SLUE & 0.52 & 0.24 & 0.33 \\ \cline{2-6}  & PhoBERT\_large & XLSR-53-Viet & SLUE & 0.46 & 0.15 & 0.23 \\ \cline{2-6}  & WhoBERT\_large & WLSR-53-Viet & Mod. SLUE & 0.55 & 0.18 & 0.27 \\ \cline{2-6}  & WhoBERT\_large & w2v2-Viet & SLUE & 0.45 & 0.15 & 0.22 \\ \cline{2-6}  & WhoBERT\_large & w2v2-Viet & Mod. SLUE & 0.55 & 0.18 & 0.27 \\ \cline{2-6}  & XLM-R\_base & XLSR-53-Viet & SLUE & 0.38 & 0.13 & 0.19 \\ \cline{2-6}  & XLM-R\_base & XLSR-53-Viet & Mod. SLUE & 0.47 & 0.16 & 0.24 \\ \cline{2-6}  & XMM-R\_base & w2v2-Viet & SLUE & 0.34 & 0.12 & 0.17 \\ \cline{2-6}  & XMM-R\_base & w2v2-Viet & Mod. SLUE & 0.44 & 0.15 & 0.22 \\ \cline{2-6}  & XMM-R\_large & XLSR-53-Viet & SLUE & 0.47 & 0.15 & 0.23 \\ \cline{2-6}  & XMM-R\_large & XLSR-53-Viet & Mod. SLUE & 0.55 & 0.18 & 0.27 \\ \cline{2-6}  & XMM-R\_large & w2v2-Viet & SLUE & 0.43 & 0.13 & 0.20 \\ \cline{2-6}  & XMM-R\_large & w2v2-Viet & Mod. SLUE & 0.53 & 0.16 & 0.25 \\ \hline \end{tabular}
\end{table}
Table 33: NER results of **MEDDEVICETECHNIQUE** entity type (in percent) on **ASR output** of test set using various pre-trained language models and ASR models. Metrics shown are Precision, Recall, and overall micro/macro F1 score. Evaluation toolkits used are: SLUE and our modified SLUE.





\begin{table}
\begin{tabular}{|c|l|l|l|l|l|l|} \hline
**Entity Type** & **NER** & **ASR** & **Eval. Toolkit** & **Prec.** & **Rec.** & **F1** \\ \hline  & ViDeBERTa\_base & XLSR-53-Viet & SLUE & 0.76 & 0.75 & 0.76 \\ \cline{2-8}  & ViDeBERTa\_base & XLSR-53-Viet & Mod. SLUE & 0.78 & 0.76 & 0.77 \\ \cline{2-8}  & ViDeBERTa\_base & w2v2-Viet & SLUE & 0.77 & 0.74 & 0.76 \\ \cline{2-8}  & ViDeBERTa\_base & w2v2-Viet & Mod. SLUE & 0.79 & 0.75 & 0.77 \\ \cline{2-8}  & ViT5\_base & XLSR-53-Viet & SLUE & 0.78 & 0.68 & 0.73 \\ \cline{2-8}  & ViT5\_base & XLSR-53-Viet & Mod. SLUE & 0.79 & 0.70 & 0.74 \\ \cline{2-8}  & ViT5\_base & w2v2-Viet & SLUE & 0.79 & 0.69 & 0.74 \\ \cline{2-8}  & ViT5\_base & w2v2-Viet & Mod. SLUE & 0.80 & 0.70 & 0.75 \\ \cline{2-8}  & mBART-50 & XLSR-53-Viet & SLUE & 0.52 & 0.02 & 0.04 \\ \cline{2-8}  & mBART-50 & XLSR-53-Viet & Mod. SLUE & 0.54 & 0.02 & 0.05 \\ \cline{2-8}  & mBART-50 & w2v2-Viet & SLUE & 0.57 & 0.03 & 0.05 \\ \cline{2-8}  & mBART-50 & w2v2-Viet & Mod. SLUE & 0.60 & 0.03 & 0.06 \\ \cline{2-8}  & BARTpho & XLSR-53-Viet & SLUE & 0.79 & 0.78 & 0.79 \\ \cline{2-8}  & BARTpho & XLSR-53-Viet & Mod. SLUE & 0.80 & 0.79 & 0.80 \\ \cline{2-8}  & BARTpho & w2v2-Viet & SLUE & 0.79 & 0.77 & 0.78 \\ \cline{2-8}  & BARTpho & w2v2-Viet & Mod. SLUE & 0.81 & 0.79 & 0.80 \\ \cline{2-8}  & PhoBERT\_base-v2 & XLSR-53-Viet & SLUE & 0.79 & 0.84 & 0.81 \\ \cline{2-8} OCCUPATION & PhoBERT\_base-v2 & XLSR-53-Viet & Mod. SLUE & 0.80 & 0.85 & 0.82 \\ \cline{2-8}  & PhoBERT\_base-v2 & w2v2-Viet & SLUE & 0.81 & 0.84 & 0.82 \\ \cline{2-8}  & PhoBERT\_base-v2 & w2v2-Viet & Mod. SLUE & 0.82 & 0.84 & 0.83 \\ \cline{2-8}  & PhoBERT\_base & XLSR-53-Viet & SLUE & 0.79 & 0.84 & 0.81 \\ \cline{2-8}  & PhoBERT\_base & XLSR-53-Viet & Mod. SLUE & 0.79 & 0.85 & 0.82 \\ \cline{2-8}  & PhoBERT\_base & w2v2-Viet & SLUE & 0.80 & 0.84 & 0.82 \\ \cline{2-8}  & PhoBERT\_large & XLRS-53-Viet & SLUE & 0.81 & 0.85 & 0.83 \\ \cline{2-8}  & PhoBERT\_large & w2v2-Viet & SLUE & 0.82 & 0.84 & 0.83 \\ \cline{2-8}  & WhoBERT\_large & w2v2-Viet & Mod. SLUE & 0.82 & 0.85 & 0.83 \\ \cline{2-8}  & XLM-R\_base & XLSR-53-Viet & SLUE & 0.74 & 0.83 & 0.79 \\ \cline{2-8}  & XLM-R\_base & XLSR-53-Viet & Mod. SLUE & 0.75 & 0.85 & 0.80 \\ \cline{2-8}  & XLM-R\_base & w2v2-Viet & SLUE & 0.76 & 0.83 & 0.79 \\ \cline{2-8}  & XMM-R\_base & w2v2-Viet & Mod. SLUE & 0.77 & 0.83 & 0.80 \\ \cline{2-8}  & XLM-R\_large & XLSR-53-Viet & SLUE & 0.81 & 0.84 & 0.82 \\ \cline{2-8}  & XLM-R\_large & XLSR-53-Viet & Mod. SLUE & 0.81 & 0.85 & 0.83 \\ \cline{2-8}  & XLM-R\_large & w2v2-Viet & SLUE & 0.82 & 0.84 & 0.83 \\ \cline{2-8}  & XMM-R\_large & w2v2-Viet & Mod. SLUE & 0.82 & 0.84 & 0.83 \\ \hline \end{tabular}
\end{table}
Table 34: NER results of **OCCUPATION** entity type (in percent) on **ASR output** of test set using various pre-trained language models and ASR models. Metrics shown are Precision, Recall, and overall micro/macro F1 score. Evaluation toolkits used are: SLUE and our modified SLUE.





\begin{table}
\begin{tabular}{|c|l|l|l|l|l|l|} \hline
**Entity Type** & **NER** & **ASR** & **Eval. Toolkit** & **Prec.** & **Rec.** & **F1** \\ \hline  & ViDeBERTa base & XLSR-53-Viet & SLUE & 0.36 & 0.31 & 0.33 \\ \cline{2-7}  & ViDeBERTa base & XLSR-53-Viet & Mod. SLUE & 0.49 & 0.41 & 0.45 \\ \cline{2-7}  & ViDeBERTa base & w2v2-Viet & SLUE & 0.37 & 0.31 & 0.34 \\ \cline{2-7}  & ViDeBERTa base & w2v2-Viet & Mod. SLUE & 0.50 & 0.42 & 0.46 \\ \cline{2-7}  & ViT5\_base & XLSR-53-Viet & SLUE & 0.50 & 0.39 & 0.44 \\ \cline{2-7}  & ViT5\_base & XLSR-53-Viet & Mod. SLUE & 0.59 & 0.46 & 0.52 \\ \cline{2-7}  & ViT5\_base & w2v2-Viet & SLUE & 0.53 & 0.40 & 0.46 \\ \cline{2-7}  & ViT5\_base & w2v2-Viet & Mod. SLUE & 0.64 & 0.48 & 0.55 \\ \cline{2-7}  & mBART-50 & XLSR-53-Viet & SLUE & 0.29 & 0.06 & 0.09 \\ \cline{2-7}  & mBART-50 & XLSR-53-Viet & Mod. SLUE & 0.46 & 0.09 & 0.15 \\ \cline{2-7}  & mBART-50 & w2v2-Viet & SLUE & 0.30 & 0.06 & 0.10 \\ \cline{2-7}  & mBART-50 & w2v2-Viet & Mod. SLUE & 0.42 & 0.08 & 0.14 \\ \cline{2-7}  & BARTpho & XLSR-53-Viet & SLUE & 0.52 & 0.42 & 0.46 \\ \cline{2-7}  & BARTpho & XLSR-53-Viet & Mod. SLUE & 0.62 & 0.50 & 0.55 \\ \cline{2-7}  & BARTpho & w2v2-Viet & SLUE & 0.53 & 0.42 & 0.47 \\ \cline{2-7}  & BARTpho & w2v2-Viet & Mod. SLUE & 0.63 & 0.50 & 0.56 \\ \cline{2-7}  & PhoBERT\_base-v2 & XLSR-53-Viet & SLUE & 0.53 & 0.48 & 0.50 \\ \cline{2-7} ORGAN & PhoBERT\_base-v2 & XLSR-53-Viet & Mod. SLUE & 0.62 & 0.56 & 0.59 \\ \cline{2-7}  & PhoBERT\_base-v2 & w2v2-Viet & SLUE & 0.55 & 0.49 & 0.52 \\ \cline{2-7}  & PhoBERT\_base-v2 & w2v2-Viet & Mod. SLUE & 0.64 & 0.57 & 0.60 \\ \cline{2-7}  & PhoBERT\_base & XLSR-53-Viet & SLUE & 0.53 & 0.47 & 0.50 \\ \cline{2-7}  & PhoBERT\_base & XLSR-53-Viet & Mod. SLUE & 0.63 & 0.56 & 0.59 \\ \cline{2-7}  & WhoBERT\_base & w2v2-Viet & SLUE & 0.54 & 0.48 & 0.51 \\ \cline{2-7}  & PhoBERT\_base & w2v2-Viet & Mod. SLUE & 0.63 & 0.56 & 0.59 \\ \cline{2-7}  & WhoBERT\_large & XLSR-53-Viet & SLUE & 0.52 & 0.46 & 0.49 \\ \cline{2-7}  & ThoSBERT\_large & XLSR-53-Viet & Mod. SLUE & 0.62 & 0.55 & 0.58 \\ \cline{2-7}  & PhoBERT\_large & w2v2-Viet & SLUE & 0.53 & 0.48 & 0.50 \\ \cline{2-7}  & WhoBERT\_large & w2v2-Viet & Mod. SLUE & 0.63 & 0.57 & 0.59 \\ \cline{2-7}  & XLM-R\_base & XLSR-53-Viet & SLUE & 0.52 & 0.48 & 0.50 \\ \cline{2-7}  & XLM-R\_base & XLSR-53-Viet & Mod. SLUE & 0.61 & 0.56 & 0.59 \\ \cline{2-7}  & XLM-R\_base & w2v2-Viet & SLUE & 0.53 & 0.50 & 0.51 \\ \cline{2-7}  & XLM-R\_base & w2v2-Viet & Mod. SLUE & 0.62 & 0.58 & 0.60 \\ \cline{2-7}  & XLM-R\_large & XLSR-53-Viet & SLUE & 0.55 & 0.47 & 0.50 \\ \cline{2-7}  & XLM-R\_large & XLSR-53-Viet & Mod. SLUE & 0.64 & 0.55 & 0.59 \\ \cline{2-7}  & XLM-R\_large & w2v2-Viet & SLUE & 0.56 & 0.48 & 0.52 \\ \cline{2-7}  & XLM-R\_large & w2v2-Viet & Mod. SLUE & 0.65 & 0.56 & 0.60 \\ \hline \end{tabular}
\end{table}
Table 35: NER results of **ORGAN** entity type (in percent) on **ASR output** of test set using various pre-trained language models and ASR models. Metrics shown are Precision, Recall, and overall micro/macro F1 score. Evaluation toolkits used are: SLUE and our modified SLUE.





\begin{table}
\begin{tabular}{|c|l|l||l|r|r|r|} \hline
**Entity Type** & **NER** & **ASR** & **Eval. Toolkit** & **Prec.** & **Rec.** & **F1** \\ \hline  & ViDeBERTa\_base & XLSR-53-Viet & SLUE & 0.00 & 0.00 & 0.00 \\ \cline{2-6}  & ViDeBERTa\_base & XLSR-53-Viet & Mod. SLUE & 0.00 & 0.00 & 0.00 \\ \cline{2-6}  & ViDeBERTa\_base & w2v2-Viet & SLUE & 0.00 & 0.00 & 0.00 \\ \cline{2-6}  & ViDeBERTa\_base & w2v2-Viet & Mod. SLUE & 0.00 & 0.00 & 0.00 \\ \cline{2-6}  & ViT5\_base & XLSR-53-Viet & SLUE & 0.88 & 0.30 & 0.45 \\ \cline{2-6}  & ViT5\_base & XLSR-53-Viet & Mod. SLUE & 0.88 & 0.30 & 0.45 \\ \cline{2-6}  & ViT5\_base & w2v2-Viet & SLUE & 0.81 & 0.27 & 0.41 \\ \cline{2-6}  & ViT5\_base & w2v2-Viet & Mod. SLUE & 0.81 & 0.27 & 0.41 \\ \cline{2-6}  & mBART-50 & XLSR-53-Viet & SLUE & 0.00 & 0.00 & 0.00 \\ \cline{2-6}  & mBART-50 & XLSR-53-Viet & Mod. SLUE & 0.00 & 0.00 & 0.00 \\ \cline{2-6}  & mBART-50 & w2v2-Viet & SLUE & 0.00 & 0.00 & 0.00 \\ \cline{2-6}  & mBART-50 & w2v2-Viet & Mod. SLUE & 0.00 & 0.00 & 0.00 \\ \cline{2-6}  & BARTPho & XLSR-53-Viet & SLUE & 0.78 & 0.34 & 0.47 \\ \cline{2-6}  & BARTpho & XLSR-53-Viet & Mod. SLUE & 0.82 & 0.36 & 0.50 \\ \cline{2-6}  & BARTPho & w2v2-Viet & SLUE & 0.76 & 0.36 & 0.48 \\ \cline{2-6}  & BARTpho & w2v2-Viet & Mod. SLUE & 0.82 & 0.39 & 0.53 \\ \cline{2-6}  & PhoBERT\_base-v2 & XLSR-53-Viet & SLUE & 0.75 & 0.56 & 0.64 \\ \cline{2-6}  & PhoBERT\_base-v2 & XLSR-53-Viet & Mod. SLUE & 0.78 & 0.59 & 0.67 \\ \cline{2-6}  & PhoBERT\_base-v2 & w2v2-Viet & SLUE & 0.76 & 0.55 & 0.64 \\ \cline{2-6}  & PhoBERT\_base-v2 & w2v2-Viet & Mod. SLUE & 0.79 & 0.57 & 0.66 \\ \cline{2-6}  & WhoBERT\_base & XLSR-53-Viet & SLUE & 0.65 & 0.49 & 0.56 \\ \cline{2-6}  & WhoBERT\_base & XLSR-53-Viet & Mod. SLUE & 0.66 & 0.49 & 0.56 \\ \cline{2-6}  & WhoBERT\_base & w2v2-Viet & SLUE & 0.62 & 0.45 & 0.52 \\ \cline{2-6}  & PhoBERT\_base & w2v2-Viet & Mod. SLUE & 0.63 & 0.46 & 0.53 \\ \cline{2-6}  & WhoBERT\_large & XLSR-53-Viet & SLUE & 0.73 & 0.50 & 0.59 \\ \cline{2-6}  & WhoBERT\_large & XLSR-53-Viet & Mod. SLUE & 0.73 & 0.50 & 0.60 \\ \cline{2-6}  & WhoBERT\_large & w2v2-Viet & SLUE & 0.69 & 0.49 & 0.57 \\ \cline{2-6}  & WhoBERT\_large & w2v2-Viet & Mod. SLUE & 0.71 & 0.50 & 0.59 \\ \cline{2-6}  & XLM-R\_base & XLSR-53-Viet & SLUE & 0.62 & 0.26 & 0.37 \\ \cline{2-6}  & XLM-R\_base & XLSR-53-Viet & Mod. SLUE & 0.64 & 0.27 & 0.38 \\ \cline{2-6}  & XMM-R\_base & w2v2-Viet & SLUE & 0.57 & 0.26 & 0.35 \\ \cline{2-6}  & XLM-R\_base & w2v2-Viet & Mod. SLUE & 0.59 & 0.27 & 0.37 \\ \cline{2-6}  & XLM-R\_large & XLSR-53-Viet & SLUE & 0.66 & 0.56 & 0.61 \\ \cline{2-6}  & XLM-R\_large & XLSR-53-Viet & Mod. SLUE & 0.68 & 0.58 & 0.63 \\ \cline{2-6}  & XMM-R\_large & w2v2-Viet & SLUE & 0.65 & 0.55 & 0.60 \\ \cline{2-6}  & XLM-R\_large & w2v2-Viet & Mod. SLUE & 0.69 & 0.58 & 0.63 \\ \hline \end{tabular}
\end{table}
Table 36: NER results of **ORGANIZATION** entity type (in percent) on **ASR output** of test set using various pre-trained language models and ASR models. Metrics shown are Precision, Recall, and overall micro/macro F1 score. Evaluation toolkits used are: SLUE and our modified SLUE.





\begin{table}
\begin{tabular}{|c|l|l|l|l|l|l|} \hline
**Entity Type** & **NER** & **ASR** & **Eval. Toolkit** & **Prec.** & **Rec.** & **F1** \\ \hline  & ViDeBERTa\_base & XLSR-53-Viet & SLUE & 0.20 & 0.67 & 0.30 \\ \cline{2-8}  & ViDeBERTa\_base & XLSR-53-Viet & Mod. SLUE & 0.20 & 0.70 & 0.32 \\ \cline{2-8}  & ViDeBERTa\_base & w2v2-Viet & SLUE & 0.20 & 0.67 & 0.31 \\ \cline{2-8}  & ViDeBERTa\_base & w2v2-Viet & Mod. SLUE & 0.20 & 0.69 & 0.31 \\ \cline{2-8}  & ViT5\_base & XLSR-53-Viet & SLUE & 0.17 & 0.54 & 0.26 \\ \cline{2-8}  & ViT5\_base & XLSR-53-Viet & Mod. SLUE & 0.17 & 0.57 & 0.27 \\ \cline{2-8}  & ViT5\_base & w2v2-Viet & SLUE & 0.17 & 0.53 & 0.26 \\ \cline{2-8}  & ViT5\_base & w2v2-Viet & Mod. SLUE & 0.18 & 0.57 & 0.27 \\ \cline{2-8}  & mBART-50 & XLSR-53-Viet & SLUE & 0.03 & 0.02 & 0.02 \\ \cline{2-8}  & mBART-50 & XLSR-53-Viet & Mod. SLUE & 0.06 & 0.05 & 0.06 \\ \cline{2-8}  & mBART-50 & w2v2-Viet & SLUE & 0.05 & 0.04 & 0.04 \\ \cline{2-8}  & mBART-50 & w2v2-Viet & Mod. SLUE & 0.05 & 0.05 & 0.05 \\ \cline{2-8}  & BARTrho & XLSR-53-Viet & SLUE & 0.19 & 0.63 & 0.29 \\ \cline{2-8}  & BARTPho & XLSR-53-Viet & Mod. SLUE & 0.19 & 0.65 & 0.30 \\ \cline{2-8}  & BARTPho & w2v2-Viet & SLUE & 0.18 & 0.63 & 0.28 \\ \cline{2-8}  & BARTpho & w2v2-Viet & Mod. SLUE & 0.19 & 0.64 & 0.29 \\ \cline{2-8}  & PhoBERT\_base-v2 & XLSR-53-Viet & SLUE & 0.18 & 0.71 & 0.29 \\ \cline{2-8}  & PhoBERT\_base-v2 & XLSR-53-Viet & Mod. SLUE & 0.19 & 0.73 & 0.30 \\ \cline{2-8}  & PhoBERT\_base-v2 & w2v2-Viet & SLUE & 0.18 & 0.70 & 0.29 \\ \cline{2-8}  & PhoBERT\_base-v2 & w2v2-Viet & Mod. SLUE & 0.18 & 0.71 & 0.29 \\ \cline{2-8}  & PhoBERT\_base & XLSR-53-Viet & SLUE & 0.19 & 0.69 & 0.30 \\ \cline{2-8}  & PhoBERT\_base & XLSR-53-Viet & Mod. SLUE & 0.20 & 0.71 & 0.31 \\ \cline{2-8}  & WhoBERT\_base & w2v2-Viet & SLUE & 0.20 & 0.69 & 0.31 \\ \cline{2-8}  & PhoBERT\_base & w2v2-Viet & Mod. SLUE & 0.20 & 0.70 & 0.32 \\ \cline{2-8}  & PhoBERT\_large & XLSR-53-Viet & SLUE & 0.20 & 0.70 & 0.32 \\ \cline{2-8}  & PhoBERT\_large & XLSR-53-Viet & Mod. SLUE & 0.21 & 0.71 & 0.32 \\ \cline{2-8}  & PhoBERT\_large & w2v2-Viet & SLUE & 0.20 & 0.68 & 0.31 \\ \cline{2-8}  & WhoBERT\_large & w2v2-Viet & Mod. SLUE & 0.21 & 0.69 & 0.32 \\ \cline{2-8}  & XLM-R\_base & XLSR-53-Viet & SLUE & 0.22 & 0.70 & 0.34 \\ \cline{2-8}  & XLM-R\_base & XLSR-53-Viet & Mod. SLUE & 0.23 & 0.73 & 0.35 \\ \cline{2-8}  & XMM-R\_base & w2v2-Viet & SLUE & 0.23 & 0.70 & 0.34 \\ \cline{2-8}  & XLM-R\_base & w2v2-Viet & Mod. SLUE & 0.23 & 0.72 & 0.35 \\ \cline{2-8}  & XLM-R\_large & XLSR-53-Viet & SLUE & 0.24 & 0.70 & 0.36 \\ \cline{2-8}  & XLM-R\_large & XLSR-53-Viet & Mod. SLUE & 0.25 & 0.71 & 0.37 \\ \cline{2-8}  & XLM-R\_large & w2v2-Viet & SLUE & 0.25 & 0.69 & 0.37 \\ \cline{2-8}  & XLM-R\_large & w2v2-Viet & Mod. SLUE & 0.26 & 0.70 & 0.38 \\ \hline \end{tabular}
\end{table}
Table 37: NER results of **PERSONALCARE** entity type (in percent) on **ASR output** of test set using various pre-trained language models and ASR models. Metrics shown are Precision, Recall, and overall micro/macro F1 score. Evaluation toolkits used are: SLUE and our modified SLUE.





\begin{table}
\begin{tabular}{|c|l|l|l|l|l|l|} \hline
**Entity Type** & **NER** & **ASR** & **Eval. Toolkit** & **Prec.** & **Rec.** & **F1** \\ \hline  & ViDeBERTa\_base & XLSR-53-Viet & SLUE & 0.00 & 0.00 & 0.00 \\ \cline{2-6}  & ViDeBERTa\_base & XLSR-53-Viet & Mod. SLUE & 0.00 & 0.06 & 0.01 \\ \cline{2-6}  & ViDeBERTa\_base & w2v2-Viet & SLUE & 0.00 & 0.00 & 0.00 \\ \cline{2-6}  & ViDeBERTa\_base & w2v2-Viet & Mod. SLUE & 0.00 & 0.06 & 0.01 \\ \cline{2-6}  & ViT5\_base & XLSR-53-Viet & SLUE & 0.01 & 0.11 & 0.01 \\ \cline{2-6}  & ViT5\_base & XLSR-53-Viet & Mod. SLUE & 0.01 & 0.14 & 0.02 \\ \cline{2-6}  & ViT5\_base & w2v2-Viet & SLUE & 0.01 & 0.19 & 0.03 \\ \cline{2-6}  & ViT5\_base & w2v2-Viet & Mod. SLUE & 0.02 & 0.22 & 0.03 \\ \cline{2-6}  & mBART-50 & XLSR-53-Viet & SLUE & 0.00 & 0.00 & 0.00 \\ \cline{2-6}  & mBART-50 & XLSR-53-Viet & Mod. SLUE & 0.00 & 0.00 & 0.00 \\ \cline{2-6}  & mBART-50 & w2v2-Viet & SLUE & 0.00 & 0.00 & 0.00 \\ \cline{2-6}  & mBART-50 & w2v2-Viet & Mod. SLUE & 0.00 & 0.00 & 0.00 \\ \cline{2-6}  & BARTPho & XLSR-53-Viet & SLUE & 0.00 & 0.06 & 0.01 \\ \cline{2-6}  & BARTPho & XLSR-53-Viet & Mod. SLUE & 0.01 & 0.19 & 0.02 \\ \cline{2-6}  & BARTPho & w2v2-Viet & SLUE & 0.00 & 0.06 & 0.01 \\ \cline{2-6}  & BARTPho & XLSR-53-Viet & Mod. SLUE & 0.01 & 0.19 & 0.02 \\ \cline{2-6}  & BROBERTbase-v2 & XLSR-53-Viet & SLUE & 0.01 & 0.14 & 0.01 \\ \cline{2-6} PREVENTIVEMED & PhoBERT\_base-v2 & XLSR-53-Viet & Mod. SLUE & 0.01 & 0.22 & 0.02 \\ \cline{2-6}  & PhoBERT\_base-v2 & w2v2-Viet & SLUE & 0.01 & 0.19 & 0.02 \\ \cline{2-6}  & PhoBERT\_base-v2 & w2v2-Viet & Mod. SLUE & 0.01 & 0.28 & 0.03 \\ \cline{2-6}  & PhoBERT\_base & XLSR-53-Viet & SLUE & 0.01 & 0.22 & 0.02 \\ \cline{2-6}  & WhoBERT\_base & XLSR-53-Viet & Mod. SLUE & 0.01 & 0.28 & 0.03 \\ \cline{2-6}  & PhoBERT\_base & w2v2-Viet & SLUE & 0.01 & 0.28 & 0.03 \\ \cline{2-6}  & PhoBERT\_base & w2v2-Viet & Mod. SLUE & 0.01 & 0.28 & 0.03 \\ \cline{2-6}  & WhoBERT\_base & w2v2-Viet & SLUE & 0.01 & 0.28 & 0.03 \\ \cline{2-6}  & WhoBERT\_base & WLSR-53-Viet & SLUE & 0.00 & 0.00 & 0.00 \\ \cline{2-6}  & XLM-R\_base & XLSR-53-Viet & Mod. SLUE & 0.00 & 0.06 & 0.01 \\ \cline{2-6}  & XLM-R\_base & w2v2-Viet & SLUE & 0.00 & 0.00 & 0.00 \\ \cline{2-6}  & XLM-R\_base & w2v2-Viet & Mod. SLUE & 0.00 & 0.06 & 0.01 \\ \cline{2-6}  & XLM-R\_large & XLSR-53-Viet & SLUE & 0.01 & 0.11 & 0.01 \\ \cline{2-6}  & XLM-R\_large & XLSR-53-Viet & Mod. SLUE & 0.01 & 0.19 & 0.02 \\ \cline{2-6}  & XLM-R\_large & w2v2-Viet & SLUE & 0.01 & 0.14 & 0.01 \\ \cline{2-6}  & XLM-R\_large & w2v2-Viet & Mod. SLUE & 0.01 & 0.22 & 0.02 \\ \hline \end{tabular}
\end{table}
Table 38: NER results of **PREVENTIVEMD** entity type (in percent) on **ASR output** of test set using various pre-trained language models and ASR models. Metrics shown are Precision, Recall, and overall micro/macro F1 score. Evaluation toolkits used are: SLUE and our modified SLUE.





\begin{table}
\begin{tabular}{|c|l|l|l|l|l|l|} \hline
**Entity Type** & **NER** & **ASR** & **Eval. Toolkit** & **Prec.** & **Rec.** & **F1** \\ \hline  & ViDeBERTa base & XLSR-53-Viet & SLUE & 0.56 & 0.24 & 0.33 \\ \cline{2-7}  & ViDeBERTa base & XLSR-53-Viet & Mod. SLUE & 0.61 & 0.26 & 0.36 \\ \cline{2-7}  & ViDeBERTa base & w2v2-Viet & SLUE & 0.57 & 0.24 & 0.34 \\ \cline{2-7}  & ViDeBERTa base & w2v2-Viet & Mod. SLUE & 0.62 & 0.26 & 0.36 \\ \cline{2-7}  & ViT5\_base & XLSR-53-Viet & SLUE & 0.46 & 0.29 & 0.36 \\ \cline{2-7}  & ViT5\_base & XLSR-53-Viet & Mod. SLUE & 0.51 & 0.32 & 0.39 \\ \cline{2-7}  & ViT5\_base & w2v2-Viet & SLUE & 0.51 & 0.28 & 0.36 \\ \cline{2-7}  & ViT5\_base & w2v2-Viet & Mod. SLUE & 0.57 & 0.32 & 0.41 \\ \cline{2-7}  & mBART-50 & XLSR-53-Viet & SLUE & 0.22 & 0.05 & 0.08 \\ \cline{2-7}  & mBART-50 & XLSR-53-Viet & Mod. SLUE & 0.32 & 0.07 & 0.11 \\ \cline{2-7}  & mBART-50 & w2v2-Viet & SLUE & 0.22 & 0.06 & 0.09 \\ \cline{2-7}  & mBART-50 & w2v2-Viet & Mod. SLUE & 0.31 & 0.08 & 0.13 \\ \cline{2-7}  & BARTpho & XLSR-53-Viet & SLUE & 0.52 & 0.29 & 0.37 \\ \cline{2-7}  & BARTpho & XLSR-53-Viet & Mod. SLUE & 0.59 & 0.33 & 0.42 \\ \cline{2-7}  & BARTpho & w2v2-Viet & SLUE & 0.54 & 0.29 & 0.38 \\ \cline{2-7}  & BARTpho & w2v2-Viet & Mod. SLUE & 0.61 & 0.33 & 0.43 \\ \cline{2-7}  & PhoBERT\_base-v2 & XLSR-53-Viet & SLUE & 0.57 & 0.29 & 0.39 \\ \cline{2-7}  & PhoBERT\_base-v2 & XLSR-53-Viet & Mod. SLUE & 0.62 & 0.32 & 0.42 \\ \cline{2-7}  & PhoBERT\_base-v2 & w2v2-Viet & SLUE & 0.60 & 0.32 & 0.42 \\ \cline{2-7}  & PhoBERT\_base-v2 & w2v2-Viet & Mod. SLUE & 0.65 & 0.35 & 0.46 \\ \cline{2-7}  & PhoBERT\_base & XLSR-53-Viet & SLUE & 0.53 & 0.26 & 0.35 \\ \cline{2-7}  & PhoBERT\_base & XLSR-53-Viet & Mod. SLUE & 0.56 & 0.27 & 0.37 \\ \cline{2-7}  & PhoBERT\_base & w2v2-Viet & SLUE & 0.56 & 0.27 & 0.37 \\ \cline{2-7}  & PhoBERT\_base & w2v2-Viet & Mod. SLUE & 0.61 & 0.30 & 0.40 \\ \cline{2-7}  & WhoBERT\_large & XLSR-53-Viet & SLUE & 0.57 & 0.28 & 0.38 \\ \cline{2-7}  & PhoBERT\_large & XLSR-53-Viet & Mod. SLUE & 0.61 & 0.31 & 0.41 \\ \cline{2-7}  & PhoBERT\_large & w2v2-Viet & SLUE & 0.61 & 0.28 & 0.39 \\ \cline{2-7}  & WhoBERT\_large & w2v2-Viet & Mod. SLUE & 0.66 & 0.31 & 0.42 \\ \cline{2-7}  & XLM-R\_base & XLSR-53-Viet & SLUE & 0.50 & 0.32 & 0.39 \\ \cline{2-7}  & XLM-R\_base & XLSR-53-Viet & Mod. SLUE & 0.54 & 0.35 & 0.42 \\ \cline{2-7}  & XLM-R\_base & w2v2-Viet & SLUE & 0.54 & 0.33 & 0.41 \\ \cline{2-7}  & XLM-R\_base & w2v2-Viet & Mod. SLUE & 0.58 & 0.35 & 0.44 \\ \cline{2-7}  & XLM-R\_large & XLSR-53-Viet & SLUE & 0.58 & 0.28 & 0.38 \\ \cline{2-7}  & XLM-R\_large & XLSR-53-Viet & Mod. SLUE & 0.62 & 0.30 & 0.41 \\ \cline{2-7}  & XLM-R\_large & w2v2-Viet & SLUE & 0.62 & 0.30 & 0.40 \\ \cline{2-7}  & XLM-R\_large & w2v2-Viet & Mod. SLUE & 0.65 & 0.32 & 0.43 \\ \hline \end{tabular}
\end{table}
Table 39: NER results of **SURGERY** entity type (in percent) on **ASR output** of test set using various pre-trained language models and ASR models. Metrics shown are Precision, Recall, and overall micro/macro F1 score. Evaluation toolkits used are: SLUE and our modified SLUE.





\begin{table}
\begin{tabular}{|c|l|l|l|l|l|l|} \hline
**Entity Type** & **NER** & **ASR** & **Eval. Toolkit** & **Prec.** & **Rec.** & **F1** \\ \hline  & ViDeBERTa\_base & XLSR-53-Viet & SLUE & 0.00 & 0.00 & 0.00 \\ \cline{2-6}  & ViDeBERTa\_base & XLSR-53-Viet & Mod. SLUE & 0.00 & 0.00 & 0.00 \\ \cline{2-6}  & ViDeBERTa\_base & w2v2-Viet & SLUE & 0.00 & 0.00 & 0.00 \\ \cline{2-6}  & ViDeBERTa\_base & w2v2-Viet & Mod. SLUE & 0.00 & 0.00 & 0.00 \\ \cline{2-6}  & ViT5\_base & XLSR-53-Viet & SLUE & 0.00 & 0.00 & 0.00 \\ \cline{2-6}  & ViT5\_base & XLSR-53-Viet & Mod. SLUE & 0.00 & 0.00 & 0.00 \\ \cline{2-6}  & ViT5\_base & w2v2-Viet & SLUE & 0.00 & 0.00 & 0.00 \\ \cline{2-6}  & ViT5\_base & w2v2-Viet & Mod. SLUE & 0.00 & 0.00 & 0.00 \\ \cline{2-6}  & mBART-50 & XLSR-53-Viet & SLUE & 0.00 & 0.00 & 0.00 \\ \cline{2-6}  & mBART-50 & XLSR-53-Viet & Mod. SLUE & 0.00 & 0.00 & 0.00 \\ \cline{2-6}  & mBART-50 & w2v2-Viet & SLUE & 0.00 & 0.00 & 0.00 \\ \cline{2-6}  & mBART-50 & w2v2-Viet & Mod. SLUE & 0.00 & 0.00 & 0.00 \\ \cline{2-6}  & mBART-50 & w2v2-Viet & SLUE & 0.00 & 0.00 & 0.00 \\ \cline{2-6}  & mBART-50 & w2v2-Viet & Mod. SLUE & 0.00 & 0.00 & 0.00 \\ \cline{2-6}  & mBARTpho & XLSR-53-Viet & SLUE & 0.17 & 0.50 & 0.25 \\ \cline{2-6}  & BARTpho & w2v2-Viet & SLUE & 0.00 & 0.00 & 0.00 \\ \cline{2-6}  & BARTpho & w2v2-Viet & Mod. SLUE & 0.00 & 0.00 & 0.00 \\ \cline{2-6}  & PhoBERT\_base-v2 & XLSR-53-Viet & SLUE & 0.50 & 1,00 & 0.67 \\ \cline{2-6}  & PhoBERT\_base-v2 & XLSR-53-Viet & Mod. SLUE & 0.50 & 1,00 & 0.67 \\ \cline{2-6}  & PnoBERT\_base-v2 & W2v2-Viet & SLUE & 0.00 & 0.00 & 0.00 \\ \cline{2-6}  & PhoBERT\_base-v2 & w2v2-Viet & SLUE & 0.00 & 0.00 & 0.00 \\ \cline{2-6}  & PhoBERT\_base-v2 & w2v2-Viet & Mod. SLUE & 0.00 & 0.00 & 0.00 \\ \cline{2-6}  & PhoBERT\_base-v2 & w2v2-Viet & Mod. SLUE & 0.00 & 0.00 & 0.00 \\ \cline{2-6}  & PhoBERT\_base & XLSR-53-Viet & SLUE & 0.50 & 1,00 & 0.67 \\ \cline{2-6}  & ThoBERT\_large & XLSR-53-Viet & Mod. SLUE & 0.50 & 1,00 & 0.67 \\ \cline{2-6}  & WhoBERT\_large & w2v2-Viet & SLUE & 0.00 & 0.00 & 0.00 \\ \cline{2-6}  & XLM-R\_base & XLSR-53-Viet & SLUE & 0.00 & 0.00 & 0.00 \\ \cline{2-6}  & XLM-R\_base & XLSR-53-Viet & Mod. SLUE & 0.00 & 0.00 & 0.00 \\ \cline{2-6}  & XLM-R\_base & w2v2-Viet & SLUE & 0.00 & 0.00 & 0.00 \\ \cline{2-6}  & XLM-R\_base & w2v2-Viet & Mod. SLUE & 0.00 & 0.00 & 0.00 \\ \cline{2-6}  & XLM-R\_large & XLSR-53-Viet & SLUE & 0.50 & 1,00 & 0.67 \\ \cline{2-6}  & XLM-R\_large & XLSR-53-Viet & Mod. SLUE & 0.50 & 1,00 & 0.67 \\ \cline{2-6}  & XLM-R\_large & WLXZ-Viet & SLUE & 0.00 & 0.00 & 0.00 \\ \cline{2-6}  & XLM-R\_large & w2v2-Viet & Mod. SLUE & 0.00 & 0.00 & 0.00 \\ \hline \end{tabular}
\end{table}
Table 40: NER results of **TRANSPORTATION** entity type (in percent) on **ASR output** of test set using various pre-trained language models and ASR models. Metrics shown are Precision, Recall, and overall micro/macro F1 score. Evaluation toolkits used are: SLUE and our modified SLUE.





\begin{table}
\begin{tabular}{|c|l|l|l|l|l|l|} \hline
**Entity Type** & **NER** & **ASR** & **Eval. Toolkit** & **Prec.** & **Rec.** & **F1** \\ \hline  & ViDeBERTa\_base & XLSR-53-Viet & SLUE & 0.56 & 0.77 & 0.65 \\ \cline{2-6}  & ViDeBERTa\_base & XLSR-53-Viet & Mod. SLUE & 0.57 & 0.78 & 0.66 \\ \cline{2-6}  & ViDeBERTa\_base & w2v2-Viet & SLUE & 0.58 & 0.75 & 0.65 \\ \cline{2-6}  & ViDeBERTa\_base & w2v2-Viet & Mod. SLUE & 0.58 & 0.76 & 0.66 \\ \cline{2-6}  & ViT5\_base & XLSR-53-Viet & SLUE & 0.45 & 0.62 & 0.52 \\ \cline{2-6}  & ViT5\_base & XLSR-53-Viet & Mod. SLUE & 0.46 & 0.63 & 0.53 \\ \cline{2-6}  & ViT5\_base & w2v2-Viet & SLUE & 0.44 & 0.62 & 0.52 \\ \cline{2-6}  & ViT5\_base & w2v2-Viet & Mod. SLUE & 0.45 & 0.64 & 0.53 \\ \cline{2-6}  & mBART-50 & XLSR-53-Viet & SLUE & 0.19 & 0.08 & 0.11 \\ \cline{2-6}  & mBART-50 & XLSR-53-Viet & Mod. SLUE & 0.23 & 0.09 & 0.13 \\ \cline{2-6}  & mBART-50 & w2v2-Viet & SLUE & 0.30 & 0.12 & 0.17 \\ \cline{2-6}  & mBART-50 & w2v2-Viet & Mod. SLUE & 0.34 & 0.13 & 0.19 \\ \cline{2-6}  & BARTPho & XLSR-53-Viet & SLUE & 0.56 & 0.71 & 0.62 \\ \cline{2-6}  & BARTpho & XLSR-53-Viet & Mod. SLUE & 0.57 & 0.72 & 0.64 \\ \cline{2-6}  & BARTpho & w2v2-Viet & SLUE & 0.58 & 0.71 & 0.64 \\ \cline{2-6}  & BARTpho & w2v2-Viet & Mod. SLUE & 0.60 & 0.73 & 0.66 \\ \cline{2-6}  & PhoBERT\_base-v2 & XLSR-53-Viet & SLUE & 0.54 & 0.77 & 0.64 \\ \cline{2-6}  & PhoBERT\_base-v2 & XLSR-53-Viet & Mod. SLUE & 0.55 & 0.78 & 0.65 \\ \cline{2-6}  & PhoBERT\_base-v2 & w2v2-Viet & SLUE & 0.55 & 0.76 & 0.64 \\ \cline{2-6}  & PhoBERT\_base-v2 & w2v2-Viet & Mod. SLUE & 0.56 & 0.77 & 0.65 \\ \cline{2-6}  & PhoBERT\_base & XLSR-53-Viet & SLUE & 0.56 & 0.77 & 0.65 \\ \cline{2-6}  & PhoBERT\_base & XLSR-53-Viet & Mod. SLUE & 0.58 & 0.79 & 0.67 \\ \cline{2-6}  & PhoBERT\_base & w2v2-Viet & SLUE & 0.57 & 0.77 & 0.65 \\ \cline{2-6}  & WhoBERT\_base & w2v2-Viet & Mod. SLUE & 0.58 & 0.78 & 0.66 \\ \cline{2-6}  & PhoBERT\_large & XLSR-53-Viet & SLUE & 0.54 & 0.77 & 0.64 \\ \cline{2-6}  & PhoBERT\_large & XLSR-53-Viet & Mod. SLUE & 0.55 & 0.79 & 0.65 \\ \cline{2-6}  & PhoBERT\_large & w2v2-Viet & SLUE & 0.54 & 0.76 & 0.63 \\ \cline{2-6}  & PhoBERT\_large & w2v2-Viet & Mod. SLUE & 0.55 & 0.77 & 0.64 \\ \cline{2-6}  & XLM-R\_base & XLSR-53-Viet & SLUE & 0.46 & 0.76 & 0.57 \\ \cline{2-6}  & XLM-R\_base & XLSR-53-Viet & Mod. SLUE & 0.47 & 0.78 & 0.58 \\ \cline{2-6}  & XLM-R\_base & w2v2-Viet & SLUE & 0.45 & 0.76 & 0.56 \\ \cline{2-6}  & XMM-R\_base & w2v2-Viet & Mod. SLUE & 0.46 & 0.77 & 0.57 \\ \cline{2-6}  & XLM-R\_large & XLSR-53-Viet & SLUE & 0.60 & 0.77 & 0.68 \\ \cline{2-6}  & XLM-R\_large & XLSR-53-Viet & Mod. SLUE & 0.61 & 0.78 & 0.69 \\ \cline{2-6}  & XMM-R\_large & w2v2-Viet & SLUE & 0.60 & 0.76 & 0.67 \\ \cline{2-6}  & XMM-R\_large & w2v2-Viet & Mod. SLUE & 0.61 & 0.77 & 0.68 \\ \hline \end{tabular}
\end{table}
Table 41: NER results of ** TREATMENT** entity type (in percent) on **ASR output** of test set using various pre-trained language models and ASR models. Metrics shown are Precision, Recall, and overall micro/macro F1 score. Evaluation toolkits used are: SLUE and our modified SLUE.





\begin{table}
\begin{tabular}{|c|l|l|l|l|l|l|} \hline
**Entity Type** & **NER** & **ASR** & **Eval. Toolkit** & **Prec.** & **Rec.** & **F1** \\ \hline  & ViDeBERTa\_base & XLSR-53-Viet & SLUE & 0.24 & 0.34 & 0.28 \\ \cline{2-6}  & ViDeBERTa\_base & XLSR-53-Viet & Mod. SLUE & 0.29 & 0.42 & 0.34 \\ \cline{2-6}  & ViDeBERTa\_base & w2v2-Viet & SLUE & 0.25 & 0.34 & 0.29 \\ \cline{2-6}  & ViDeBERTa\_base & w2v2-Viet & Mod. SLUE & 0.29 & 0.40 & 0.34 \\ \cline{2-6}  & ViT5\_base & XLSR-53-Viet & SLUE & 0.25 & 0.38 & 0.30 \\ \cline{2-6}  & ViT5\_base & XLSR-53-Viet & Mod. SLUE & 0.28 & 0.42 & 0.33 \\ \cline{2-6}  & ViT5\_base & w2v2-Viet & SLEUE & 0.26 & 0.38 & 0.31 \\ \cline{2-6}  & ViT5\_base & w2v2-Viet & Mod. SLUE & 0.28 & 0.42 & 0.34 \\ \cline{2-6}  & mBART-50 & XLSR-53-Viet & SLUE & 0.26 & 0.05 & 0.09 \\ \cline{2-6}  & mBART-50 & XLSR-53-Viet & Mod. SLUE & 0.41 & 0.08 & 0.14 \\ \cline{2-6}  & mBART-50 & w2v2-Viet & SLUE & 0.28 & 0.06 & 0.10 \\ \cline{2-6}  & mBART-50 & w2v2-Viet & Mod. SLUE & 0.36 & 0.08 & 0.13 \\ \cline{2-6}  & BARTPho & XLSR-53-Viet & SLUE & 0.34 & 0.41 & 0.37 \\ \cline{2-6}  & BARTpho & XLSR-53-Viet & Mod. SLUE & 0.37 & 0.44 & 0.40 \\ \cline{2-6}  & BARTPho & w2v2-Viet & SLUE & 0.35 & 0.42 & 0.39 \\ \cline{2-6}  & BARTpho & w2v2-Viet & Mod. SLUE & 0.38 & 0.45 & 0.41 \\ \cline{2-6}  & PhoBERT\_base-v2 & XLSR-53-Viet & SLUE & 0.35 & 0.55 & 0.43 \\ \cline{2-6} UNITCALIBRATOR & PhoBERT\_base-v2 & XLSR-53-Viet & Mod. SLUE & 0.40 & 0.63 & 0.49 \\ \cline{2-6}  & PhoBERT\_base-v2 & w2v2-Viet & SLUE & 0.39 & 0.55 & 0.45 \\ \cline{2-6}  & PhoBERT\_base-v2 & w2v2-Viet & Mod. SLUE & 0.42 & 0.60 & 0.50 \\ \cline{2-6}  & PhoBERT\_base & XLSR-53-Viet & SLUE & 0.34 & 0.52 & 0.41 \\ \cline{2-6}  & WhoBERT\_base & XLSR-53-Viet & Mod. SLUE & 0.39 & 0.59 & 0.47 \\ \cline{2-6}  & PhoBERT\_base & w2v2-Viet & SLUE & 0.36 & 0.53 & 0.43 \\ \cline{2-6}  & PhoBERT\_base & WLXR-53-Viet & Mod. SLUE & 0.40 & 0.59 & 0.47 \\ \cline{2-6}  & WhoBERT\_large & XLSR-53-Viet & SLUE & 0.36 & 0.54 & 0.43 \\ \cline{2-6}  & WhoBERT\_large & XLSR-53-Viet & Mod. SLUE & 0.40 & 0.60 & 0.48 \\ \cline{2-6}  & WhoBERT\_large & w2v2-Viet & SLUE & 0.38 & 0.55 & 0.45 \\ \cline{2-6}  & WhoBERT\_large & w2v2-Viet & Mod. SLUE & 0.41 & 0.59 & 0.49 \\ \cline{2-6}  & XLM-R\_base & XLSR-53-Viet & SLUE & 0.32 & 0.51 & 0.40 \\ \cline{2-6}  & XLM-R\_base & XLSR-53-Viet & Mod. SLUE & 0.36 & 0.57 & 0.44 \\ \cline{2-6}  & XLM-R\_base & w2v2-Viet & SLUE & 0.34 & 0.51 & 0.41 \\ \cline{2-6}  & XLM-R\_base & w2v2-Viet & Mod. SLUE & 0.37 & 0.55 & 0.44 \\ \cline{2-6}  & XLM-R\_large & XLSR-53-Viet & SLUE & 0.36 & 0.52 & 0.42 \\ \cline{2-6}  & XLM-R\_large & XLSR-53-Viet & Mod. SLUE & 0.40 & 0.58 & 0.47 \\ \cline{2-6}  & XLM-R\_large & w2v2-Viet & SLUE & 0.40 & 0.55 & 0.46 \\ \cline{2-6}  & XLM-R\_large & w2v2-Viet & Mod. SLUE & 0.42 & 0.58 & 0.49 \\ \hline \end{tabular}
\end{table}
Table 42: NER results of **UNITCALIBRATOR** entity type (in percent) on **ASR output** of test set using various pre-trained language models and ASR models. Metrics shown are Precision, Recall, and overall micro/macro F1 score. Evaluation toolkits used are: SLUE and our modified SLUE.





\begin{table}
\begin{tabular}{|c|l|l|l|l|l|l|} \hline
**Entity Type** & **NER** & **ASR** & **Eval. Toolkit** & **Prec.** & **Rec.** & **F1** \\ \hline  & ViDeBERTa\_base & XLSR-53-Viet & SLUE & 0.34 & 0.31 & 0.30 \\ \cline{2-6}  & ViDeBERTa\_base & XLSR-53-Viet & Mod. SLUE & 0.38 & 0.34 & 0.33 \\ \cline{2-6}  & ViDeBERTa\_base & w2v2-Viet & SLUE & 0.39 & 0.30 & 0.29 \\ \cline{2-6}  & ViDeBERTa\_base & w2v2-Viet & Mod. SLUE & 0.43 & 0.33 & 0.33 \\ \cline{2-6}  & ViT5\_base & XLSR-53-Viet & SLUE & 0.48 & 0.42 & 0.42 \\ \cline{2-6}  & ViT5\_base & XLSR-53-Viet & Mod. SLUE & 0.52 & 0.45 & 0.46 \\ \cline{2-6}  & ViT5\_base & w2v2-Viet & SLUE & 0.49 & 0.42 & 0.43 \\ \cline{2-6}  & ViT5\_base & w2v2-Viet & Mod. SLUE & 0.53 & 0.46 & 0.46 \\ \cline{2-6}  & mBART-50 & XLSR-53-Viet & SLUE & 0.28 & 0.04 & 0.07 \\ \cline{2-6}  & mBART-50 & XLSR-53-Viet & Mod. SLUE & 0.36 & 0.06 & 0.09 \\ \cline{2-6}  & mBART-50 & w2v2-Viet & SLUE & 0.29 & 0.05 & 0.08 \\ \cline{2-6}  & mBART-50 & w2v2-Viet & Mod. SLUE & 0.38 & 0.06 & 0.10 \\ \cline{2-6}  & BARTpho & XLSR-53-Viet & SLUE & 0.52 & 0.49 & 0.48 \\ \cline{2-6}  & BARTpho & XLSR-53-Viet & Mod. SLUE & 0.57 & 0.53 & 0.51 \\ \cline{2-6}  & BARTpho & w2v2-Viet & SLUE & 0.51 & 0.46 & 0.47 \\ \cline{2-6}  & BARTpho & w2v2-Viet & Mod. SLUE & 0.56 & 0.50 & 0.50 \\ \cline{2-6}  & PhoBERT\_base-v2 & XLSR-53-Viet & SLUE & 0.56 & 0.59 & 0.55 \\ \cline{2-6} Overall Macro & PhoBERT\_base-v2 & XLSR-53-Viet & Mod. SLUE & 0.59 & 0.63 & 0.58 \\ \cline{2-6}  & PhoBERT\_base-v2 & w2v2-Viet & SLUE & 0.54 & 0.53 & 0.51 \\ \cline{2-6}  & PhoBERT\_base-v2 & w2v2-Viet & Mod. SLUE & 0.57 & 0.57 & 0.54 \\ \cline{2-6}  & PhoBERT\_base & XLSR-53-Viet & SLUE & 0.54 & 0.58 & 0.53 \\ \cline{2-6}  & PhoBERT\_base & XLSR-53-Viet & Mod. SLUE & 0.57 & 0.62 & 0.56 \\ \cline{2-6}  & WhoBERT\_base & w2v2-Viet & SLUE & 0.52 & 0.52 & 0.50 \\ \cline{2-6}  & PhoBERT\_base & w2v2-Viet & Mod. SLUE & 0.55 & 0.56 & 0.53 \\ \cline{2-6}  & PhoBERT\_large & XLSR-53-Viet & SLUE & 0.56 & 0.58 & 0.54 \\ \cline{2-6}  & ChoBERT\_large & XLSR-53-Viet & Mod. SLUE & 0.59 & 0.62 & 0.57 \\ \cline{2-6}  & PhoBERT\_large & w2v2-Viet & SLUE & 0.53 & 0.53 & 0.50 \\ \cline{2-6}  & PhoBERT\_large & w2v2-Viet & Mod. SLUE & 0.57 & 0.56 & 0.53 \\ \cline{2-6}  & XLM-R\_base & XLSR-53-Viet & SLUE & 0.48 & 0.47 & 0.45 \\ \cline{2-6}  & XLM-R\_base & XLSR-53-Viet & Mod. SLUE & 0.52 & 0.51 & 0.49 \\ \cline{2-6}  & XLM-R\_base & w2v2-Viet & SLUE & 0.48 & 0.47 & 0.45 \\ \cline{2-6}  & XMM-R\_base & w2v2-Viet & Mod. SLUE & 0.51 & 0.51 & 0.49 \\ \cline{2-6}  & XLM-R\_large & XLSR-53-Viet & SLUE & 0.58 & 0.59 & 0.56 \\ \cline{2-6}  & XLM-R\_large & XLSR-53-Viet & Mod. SLUE & 0.61 & 0.62 & 0.59 \\ \cline{2-6}  & XMM-R\_large & w2v2-Viet & SLUE & 0.55 & 0.53 & 0.52 \\ \cline{2-6}  & XMM-R\_large & w2v2-Viet & Mod. SLUE & 0.58 & 0.56 & 0.55 \\ \hline \end{tabular}
\end{table}
Table 43: NER results of **Overall Macro** entity type (in percent) on **ASR output** of test set using various pre-trained language models and ASR models. Metrics shown are Precision, Recall, and overall micro/macro F1 score. Evaluation toolkits used are: SLUE and our modified SLUE.





\begin{table}
\begin{tabular}{|c|l|l|l|l|l|l|} \hline
**Entity Type** & **NER** & **ASR** & **Eval. Toolkit** & **Prec.** & **Rec.** & **F1** \\ \hline  & ViDeBERTa\_base & XLSR-53-Viet & SLUE & 0.45 & 0.34 & 0.39 \\ \cline{2-6}  & ViDeBERTa\_base & XLSR-53-Viet & Mod. SLUE & 0.51 & 0.39 & 0.45 \\ \cline{2-6}  & ViDeBERTa\_base & w2v2-Viet & SLUE & 0.45 & 0.34 & 0.39 \\ \cline{2-6}  & ViDeBERTa\_base & w2v2-Viet & Mod. SLUE & 0.51 & 0.39 & 0.44 \\ \cline{2-6}  & ViT5\_base & XLSR-53-Viet & SLUE & 0.52 & 0.46 & 0.48 \\ \cline{2-6}  & ViT5\_base & XLSR-53-Viet & Mod. SLUE & 0.58 & 0.50 & 0.54 \\ \cline{2-6}  & ViT5\_base & w2v2-Viet & SLUE & 0.53 & 0.46 & 0.49 \\ \cline{2-6}  & ViT5\_base & w2v2-Viet & Mod. SLUE & 0.59 & 0.51 & 0.54 \\ \cline{2-6}  & mBART-50 & XLSR-53-Viet & SLUE & 0.35 & 0.05 & 0.09 \\ \cline{2-6}  & mBART-50 & XLSR-53-Viet & Mod. SLUE & 0.46 & 0.07 & 0.12 \\ \cline{2-6}  & mBART-50 & w2v2-Viet & SLUE & 0.35 & 0.05 & 0.09 \\ \cline{2-6}  & mBART-50 & w2v2-Viet & Mod. SLUE & 0.47 & 0.07 & 0.12 \\ \cline{2-6}  & BARTrho & XLSR-53-Viet & SLUE & 0.56 & 0.50 & 0.53 \\ \cline{2-6}  & BARTpho & XLSR-53-Viet & Mod. SLUE & 0.61 & 0.55 & 0.58 \\ \cline{2-6}  & BARTpho & w2v2-Viet & SLUE & 0.55 & 0.50 & 0.52 \\ \cline{2-6}  & BARTpho & w2v2-Viet & Mod. SLUE & 0.61 & 0.55 & 0.58 \\ \cline{2-6}  & PhoBERT\_base-v2 & XLSR-53-Viet & SLUE & 0.57 & 0.57 & 0.57 \\ \cline{2-6} Overall Micro & PhoBERT\_base-v2 & XLSR-53-Viet & Mod. SLUE & 0.62 & 0.61 & 0.62 \\ \cline{2-6}  & PhoBERT\_base-v2 & w2v2-Viet & SLUE & 0.58 & 0.56 & 0.57 \\ \cline{2-6}  & PhoBERT\_base-v2 & w2v2-Viet & Mod. SLUE & 0.62 & 0.61 & 0.62 \\ \cline{2-6}  & PhoBERT\_base & XLSR-53-Viet & SLUE & 0.56 & 0.56 & 0.56 \\ \cline{2-6}  & WhoBERT\_base & XLSR-53-Viet & Mod. SLUE & 0.61 & 0.61 & 0.61 \\ \cline{2-6}  & WhoBERT\_base & w2v2-Viet & SLUE & 0.56 & 0.56 & 0.56 \\ \cline{2-6}  & WhoBERT\_base & w2v2-Viet & Mod. SLUE & 0.56 & 0.56 & 0.56 \\ \cline{2-6}  & WhoBERT\_base & w2v2-Viet & Mod. SLUE & 0.56 & 0.56 & 0.56 \\ \cline{2-6}  & WhoBERT\_base & w2v2-Viet & Mod. SLUE & 0.56 & 0.56 & 0.56 \\ \cline{2-6}  & WhoBERT\_large & XLSR-53-Viet & Mod. SLUE & 0.62 & 0.59 & 0.61 \\ \cline{2-6}  & XLM-R\_base & XLSR-53-Viet & SLUE & 0.54 & 0.52 & 0.53 \\ \cline{2-6}  & XLM-R\_base & XLSR-53-Viet & Mod. SLUE & 0.59 & 0.57 & 0.58 \\ \cline{2-6}  & XLM-R\_base & w2v2-Viet & SLUE & 0.54 & 0.52 & 0.53 \\ \cline{2-6}  & XMM-R\_base & w2v2-Viet & Mod. SLUE & 0.54 & 0.52 & 0.53 \\ \cline{2-6}  & XMM-R\_large & XLSR-53-Viet & SLUE & 0.59 & 0.57 & 0.58 \\ \cline{2-6}  & XMM-R\_base & XLSR-53-Viet & SLUE & 0.60 & 0.56 & 0.58 \\ \cline{2-6}  & XMM-R\_large & XLSR-53-Viet & Mod. SLUE & 0.64 & 0.60 & 0.62 \\ \cline{2-6}  & XMM-R\_large & w2v2-Viet & SLUE & 0.60 & 0.56 & 0.58 \\ \cline{2-6}  & XMM-R\_large & w2v2-Viet & Mod. SLUE & 0.64 & 0.60 & 0.62 \\ \hline \end{tabular}
\end{table}
Table 44: NER results of **Overall Micro** entity type (in percent) on **ASR output** of test set using various pre-trained language models and ASR models. Metrics shown are Precision, Recall, and overall micro/macro F1 score. Evaluation toolkits used are: SLUE and our modified SLUE.