ViMedAQA: A Vietnamese Medical Abstractive Question-Answering Dataset and Findings of Large Language Model (2024)

Abstract: Question answering involves creating answers to questions. With the growth of large language models, the ability of question-answering systems has dramatically improved. However, there is a lack of Vietnamese abstractive question-answering datasets, especially in the medical domain. Therefore, this research aims to mitigate this gap by introducing ViMedAQA 1. This Vietnamese Medical Abstractive Question-Answering dataset covers four topics in the Vietnamese medical domain, including body parts, disease, drugs, and medicine. Additionally, the empirical results on the proposed dataset examine the capability of the large language models in the Vietnamese medical domain, including reasoning, memorizing, and awareness of essential information.

heading: 1 Introduction
text: Question-answering (QA) is a Natural Language Processing (NLP) task that aims to generate an appropriate response to a given question. QA systems are categorized based on their answer format. While extractive QA systems return the sub-strings from the provided context as the answer, abstractive QA systems identify keywords within the context and then rewrite this information to answer the question. Several QA datasets are SQuAD (Rajpurkar et al., 2016) and HotpotQA (Yang et al., 2018) for extractive QA, and AQuaMuSe (Kulkarni et al., 2020) and MS MARCO (Nguyen et al., 2016) are notable abstractive QA datasets.
In the field of Vietnamese NLP, various extractive QA datasets exist, including UIT-ViQuAD Nguyen et al. (2020) and VIMQA (Le et al., 2022), both of which serve for general knowledge (opendomain QA). Within the specific context of the
* Corresponding author. 1Source code is available at: https://github.com/ trminhnam/vimedaqa and the dataset is published at: https: //huggingface.co/datasets/tmnam20/ViMedAQA.
Vietnamese medical domain, datasets such as UITViNewsQA (Van Nguyen et al., 2022) and UITViCoQA Luu et al. (2021) are available for extractive QA. However, there is a shortage of a Vietnamese abstractive question-answering corpus, especially in the medical domain.
To address the identified problem, we have developed and introduced ViMedAQA, a Vietnamese medical abstractive QA dataset. The corpus undergoes question-answer generation and human annotation stages to ensure quality while minimizing construction time. This proposed dataset is also leveraged to investigate the reasoning, denoising, and memorizing capabilities of large language models (LLMs) within the Vietnamese medical and healthcare domain. The contributions of this research work are listed as follows:
• Development of a dataset construction pipeline for abstractive QA tasks that utilizes existing LLMs to generate QA pairs from the context, thereby reducing the human effort required for question-answer creation.
• Introduction of ViMedAQA, a dedicated corpus for abstractive QA, encompasses four topics in Vietnamese medical literature: body parts, diseases, drugs, and medicine.
• Analysis of LLMs’ reasoning, critical information extracting and memorizing capabilities within the Vietnamese medical domain.

heading: 2 Related Work
text: Extractive and Abstractive QA: Extractive QA systems answer the question by extracting parts of the context (Fajcik et al., 2021). Common extractive QA datasets include SQuAD (Rajpurkar et al., 2016, 2018), Natural Questions by Kwiatkowski et al. (2019), TriviaQA by Joshi et al. (2017) and SearchQA by Dunn et al. (2017). Conversely, the abstractive QA task generates responses using the
356
model’s knowledge. When provided with context, this task becomes open-book abstractive QA, and in the absence of context, it is closed-book QA (Ciosici et al., 2021). Common datasets for abstractive QA are ELI5 by Fan et al. (2019), AQuaMuSe by Kulkarni et al. (2020), MS MARCO by Nguyen et al. (2016), PolQA by Rybak et al. (2024) and Natural Questions (Kwiatkowski et al., 2019).
Open-Domain and Close-Domain QA: Opendomain QA systems assist with general knowledge. Some common open-domain QA datasets include TriviaQA (Joshi et al., 2017), SearchQA (Dunn et al., 2017), and MS MARCO (Nguyen et al., 2016). Conversely, close-domain QA systems answer questions in specific domains such as healthcare, law, and finance. Close-domain biomedical and healthcare QA datasets include MedQuAD (Ben Abacha and Demner-Fushman, 2019), HealthQA (Zhu et al., 2019), MedMCQA (Pal et al., 2022) and BiQA (Lamurias et al., 2020).
Vietnamese QA Datasets: Multiple QA datasets have been widely published in Vietnamese. UIT-ViQuAD by Nguyen et al. (2020), which follows the SQuAD format, is constructed from Wikipedia text. VIMQA (Le et al., 2022) is a multihop extractive QA dataset based on Wikipedia. UIT-ViNewsQA, introduced by Van Nguyen et al. (2022), is built on top of Vietnamese healthcare news articles. UIT-ViCoQA, developed by Luu et al. (2021), is a medical extractive QA dataset for machine reading comprehension evaluation.

heading: 3 Dataset

heading: 3.1 Dataset Creation Process
text: The dataset creation process, visualized in Figure 1, contains three steps below.
Data source and preprocessing. Initially, raw documents are sourced from the internet. To ensure quality and credibility, we select only those written by doctors with Master’s or PhD degrees in
medicine. These documents undergo preprocessing to eliminate HTML tags, links, and non-medical content. Each document is divided into paragraphs according to the article’s structure. To respect Vietnam’s intellectual property rights, the article URL, the author’s name, and the URL are included in each paragraph. Additionally, this dataset is published for educational and research uses only.
Question-answer generation process. Using the parsed paragraph as the context, the Gemini 1.0 language model (Team et al., 2023) generates pairs of question-answer where each answer corresponding to the question must be included in the paragraph. The number of question-answer pairs to request Gemini to generate depends on the number of sentences in the paragraph as num_pairs=max{3, num_sentences_in_paragraph}.
Annotation Guideline. The team of annotators consists of five individuals (see Appendix C). Each annotator carefully evaluates the meaning and grammatical correctness of the questions and answers generated for each paragraph. They also verify whether the answer is contained within the context, either implicitly or explicitly. If any questionanswer pair is marked with a Reject label by an arbitrary labeler, the question-answer pair is removed from the dataset’s final version.
Using the outlined pipeline, we constructed and validated the ViMedAQA dataset, represented as S = {(pi, qi, ai) | 1 ≤ i ≤ n}, where n denotes the total number of samples. For each datapoint, pi denotes the paragraph, qi is the corresponding question, and ai represents the corresponding answer, with key information in the answer ai sourced directly from the corresponding paragraph pi.

heading: 3.2 Dataset Statistics
text: The dataset contains 44, 313 {p, q, a} triplets divided into train/validation/test sets. It covers four topics in the Vietnamese medical domain, includ-
ing drugs, medicine, body parts, and disease. Further information refers to Table 4 in Appendix D.
The distribution of question types is visualized in Figure 3 in Appendix D. Most questions fall under the “Open-Ended” category, totaling 40, 443, significantly outnumbering other types. The “Yes-No” category has a notable count with 3, 740 questions.

heading: 4 Methodology
text: This study assesses the capabilities of generative language models for learning, memorizing, and understanding medical information in Vietnamese.
Experiments were conducted using the proposed ViMedAQA dataset. Using an open-book QA format, the first experiment examined the model’s reasoning ability in the Vietnamese medical and healthcare domain. Each input to the model consisted of a pair, {pi, qi}, where pi is a paragraph and qi is a corresponding question. To assess the model’s ability to extract essential information to answer qi, additional unrelated paragraphs, {pj} where (j ̸= i and |{pj}| ∈ {0, 1, 2, 4, 8}), are included in the prompt. It is hypothesized that adding more noise paragraphs would degrade the model’s performance. A second experiment measured the amount of Vietnamese medical knowledge within the language models under a closed-book QA setting. The model is prompted with only the question qi and answers qi using its internal knowledge acquired during pretraining and finetuning.
We use greedy search decoding during the evaluation process to prompt the model, resulting in
highly reproducible experiments. The BLEU (Papineni et al., 2002; Lin and Och, 2004), METEOR (Banerjee and Lavie, 2005), ROUGE-L (Lin, 2004), and BERTScore (Zhang* et al., 2020) metrics are utilized to compare model’s outputs and labels.

heading: 5 Experimental Results and Analysis
text: This section reports the empirical results for the experiments following setups in Section 4.
All of the experiments use medium to small LLMs, including LLama2-7B and LLama3-7B (Touvron et al., 2023), Gemma-2B and Gemma7B by Team et al. (2023), PhoGPT-4B by Nguyen et al. (2023a), VinaLlama-7B and VinaLlama-2.7B by Nguyen et al. (2023c), and ViGPT by (Nguyen et al., 2023b). All the models are explored with the chat or instruction tuning versions.

heading: 5.1 Reasoning Ability
text: This experiment examines the models for their reasoning ability through open-book questionanswering. The results are reported in Table 1.
From Table 1, Llama3-7B outperforms other English and Vietnamese prompt template models with average (avg) scores at 57.01 and 55.05, respectively. With Vietnamese prompt, Gemma-7B gets the same mark as Llama3-7B at 55.05. Meanwhile, VinaLlama-7B achieves the highest performance across all Vietnamese LLMs, with a 58.01 average score for the English input template and a 56.89 average score for the Vietnamese input template.
In summary, Llama3-7B and VinaLlama-7B
show their strong capability in the Vietnamese medical reasoning task. Additionally, language-specific LLMs slightly outperform multilingual LLMs and using the English template results in a slight performance gain over the Vietnamese prompt template.

heading: 5.2 Awareness Ability
text: Figure 2 illustrates all model’s performance when the number of noise paragraphs (denoted as m) provided in the input prompt increases.
In both languages, the performance of most models demonstrates a shared trend. Performance gradually decreases as m increases from 0 to 2 (first stage), followed by a significant decline as k escalates from 2 to 8 (second stage). Although Llama38B exhibits the most robust performance with little noise input (m ≤ 2), the Gemma model family outperforms other models in scenarios with considerable noise (m = 8), suggesting that the Gemma model is superior in extracting crucial information amidst noisy data compared to other models.

heading: 5.3 Memorization Capability in Vietnamese Medical and Healthcare Knowledge
text: Results for this experimental scenario are presented in Table 2. In scenarios where the input prompt
only contains a question without context, the model must rely on its internal knowledge for the answer.
Among the models, VinaLlama-7B achieved the highest score of 36.80 with the English prompt template, followed closely by PhoGPT-4B with a score of 36.22. With the Vietnamese instruction template, PhoGPT-4B, scoring 36.88, outperformed all other models. The Gemma models family exhibited balanced performance across both languages, whereas the Llama family underperformed (scoring lower than 30) with the Vietnamese prompt template. These results indicate that, compared to other models, VinaLlama-7B and PhoGPT-4B possess the most extensive Vietnamese medical knowledge.

heading: 6 Conclusion
text: ViMedAQA, a Vietnamese medical abstractive question-answering dataset, is published to mitigate the lack of an abstractive QA corpus for the Vietnamese medical domain. By leveraging the available LLMs, raw question-answer pairs are automatically generated before being verified by expert annotators to ensure the dataset’s quality. Additionally, experiments to study the capability of LLMs are examined, including reasoning, memorizing, and awareness of critical information. The empirical results show that VinaLlama-7B is a large language model with powerful reasoning skills in the Vietnamese medical domain, and the Gemma model family is robust in realizing essential information across multiple noise contexts.
There are several potential directions for future work, including (1) expanding the scope of topics covered by ViMedAQA in the Vietnamese medical domain, (2) investigating the performance of LLMs in this domain under different fine-tuning methodologies, and (3) delving into the extraction
of critical data by increasing the number of incorrect paragraphs (m), and exploring solutions to mitigate performance degradation as m increases.
Limitations
Despite the introduction of ViMedAQA to address the absence of a medical abstractive QA dataset in Vietnam, this research has certain limitations.
Firstly, even though the raw documents are sourced from highly reliable resources, the LLMs occasionally fail to generate accurate questionanswer pairs. Moreover, LLMs sometimes create similar questions for different paragraphs.
Secondly, there is a lack of experiments involving fine-tuning methods on the proposed dataset for comparison with the prompting method.

heading: A Potential Risk
text: Given the rapid advancements in the medical field, the knowledge contained in ViMedAQA may become outdated. Therefore, this dataset should primarily be used to research the capabilities of QA systems in the Vietnamese medical domain rather than to develop a particular application. Any misuse of the dataset for illegal purposes is strictly prohibited. The dataset should be appropriately used to contribute to the advancements in NLP.

heading: B License
text: The raw documents are crawled from YouMed.vn2. The term of use is available on the YouMed Term Of Use webpage 3, which states that “The information included on this website is strictly for informational and educational purposes.” Hence, this research does not violate YouMed’s terms of use.
Following the YouMed term of use, the dataset is published under the Creative Commons NonCommercial 4.0 license, which requires users to use it for non-commercial purposes only.

heading: C Annotator List
text: The academic qualifications of data annotators are:
• Annotator 1 - Associate Professor in Computer Science and Comparative Linguistics.
• Annotator 2 - PhD in Computer Science and Natural Language Processing (NLP).
• Annotator 3 - PhD candidate in Comparative Linguistics.
• Annotator 4 - Undergraduate Student majoring in Computer Science and NLP.
• Annotator 5 - Undergraduate Student majoring in Computer Science and NLP

heading: D Dataset Statistics
text: Figure 3 visualizes the number of samples for each question type and subtype. Additionally, Table 3 categorizes the questions into Yes-No and OpenEnded types, with further subtypes under OpenEnded, such as Why, What, When, How, How Much/Many, Which, Who, and Where. An additional category labelled Other is included.
2https://youmed.vn/ 3https://youmed.vn/tin-tuc/term-of-use/
Table 4 shows the distribution of the dataset across training, validation, and test sets, categorized by topic. The total number of samples for each category is 44, 313, with Body part, Disease, Drug, and Medicine having 4, 970, 15, 690, 9, 780, and 13, 873 examples, respectively.
Each sample in the proposed ViMedAQA dataset has the following fields:
• question_idx: The index of the sample.
• question: The question to be answered.
• answer: The answer to the question.
• context: The context or paragraph that contains the information to answer the question.
• title: The title of the corresponding article from which the context was taken.
• keyword: The related disease/drug/body part in the question. Such as “heart attack.”
• topic: The topic of the question/context. It can be one of the following: Body part, Disease, Drug and Medicine.
• article_url: The URL of the original article.

heading: E Experiment Setup
text: In the scope of this research, the models are not trained, and their weights are not modified by gradient descent. The experiments are conducted by prompting the model directly with greedy decoding, which is more reproducible and requires less computational resources than sampling methods.
E.1 Computational Resources
The experiments are conducted on a single machine with Intel i5-14500 CPU, 32 GB RAM, and duo NVIDIA GeForce RTX 4060 Ti 16 GB cards.
E.2 Softwares
The transformers library by Wolf et al. (2020) and the datasets library by Lhoest et al. (2021) are used to load the model and datasets from HuggingFace 4, respectively. The evaluate 5 library, the bert-score framework and the rouge_score library are used to evaluate the model’s outputs.
E.3 Model Configurations
The number of parameters of the LLMs used in the experiments is reported in Table 5.
E.4 Prompt templates
The prompt templates are provided in Listing 1 for the English template. When the model does not use a system prompt (like Gemma), it is concatenated with the user prompt before feeding to the model.
4https://huggingface.co/ 5https://github.com/huggingface/evaluate
E.5 Experiment Running Time The running times of the models for the three experiment scenarios are provided in Table 6, Table 7 and Table 8. The time format is “HH:MM:DD”.

heading: Future Work
text: There are several potential directions for future work, including (1) expanding the scope of topics covered by ViMedAQA in the Vietnamese medical domain, (2) investigating the performance of LLMs in this domain under different fine-tuning methodologies, and (3) delving into the extraction
of critical data by increasing the number of incorrect paragraphs (m), and exploring solutions to mitigate performance degradation as m increases. Limitations
Despite the introduction of ViMedAQA to address the absence of a medical abstractive QA dataset in Vietnam, this research has certain limitations. Firstly, even though the raw documents are sourced from highly reliable resources, the LLMs occasionally fail to generate accurate questionanswer pairs. Moreover, LLMs sometimes create similar questions for different paragraphs. Secondly, there is a lack of experiments involving fine-tuning methods on the proposed dataset for comparison with the prompting method