# Multimodal AI and Large Language Models for Orthopantomography Radiology Report Generation and Q&A
## 4 Results  
### Results Evaluation Metrics  
Evaluating the performance of an LLM, especially in a domain-specific task, demands a combination of qualitative and quantitative metrics. Furthermore, to obtain a better understanding of the system's performance, it is necessary to assess the system's capability under different tasks at different intermediary stages. The generation of the radiology report and creating an interactive dialog with the patient in the Sinhala language as a decision support system are the two most important tasks within the purview of our study. With this scope, we have selected the metrics presented in Table 7 for the task of evaluating the system's performance. The existing objective evaluation metrics for LLMs, such as perplexity [50], and the bilingual evaluation understudy (BLEU) [51] provide insights into the quantitative performance of LLMs, while mainly focusing on language-related aspects. Furthermore, reference-based overlap utility for gisting evaluation (ROUGE) [52] measures the overlap of n-grams, word sequences, and word pairs between the generated and original text, ensuring that the generated report accurately reflects the essential content of the reference report. The BLEU metric primarily focuses on machine translation and may not effectively capture the specific content requirements of a radiology report. Perplexity measures the probability of a sequence of words and reflects the model's fluency in language generation. However, it does not directly measure content accuracy and relevance, which are vital for radiology reports. Considering the above-described reasons and the usefulness of the ROUGE metric in capturing the correct medical terminology, findings, and conclusions, we employed ROUGE to quantitatively evaluate the radiology reports. However, in the medical domain, which is rich in complex nuances and contextual understanding, relying fully on objective metrics may result in a misleading understanding \begin{table}
\begin{tabular}{c c c} \hline \hline
**Scenario** & **Qualitative Evaluation** & **Quantitative Evaluation** \\ \hline \multirow{2}{*}{Radiology report generation} & Weighted average score given by dental professionals & ROUGE \\  & Weighted average score & \\  & given by dental professionals & \\ \multirow{2}{*}{Sinhala language Q\&A in the dental domain} & Weighted average score given by dental professionals & DDSLU benchmark \\ \hline \hline \end{tabular}
\end{table} 
Table 7: Evaluation methods for each model. Figure 7: Tokenizer comparison for a sample input from the Sinhala question-answering dataset. of the model's performance. Subjective evaluation, i.e., human evaluation, is crucial in this domain. Human factors, such as domain expertise, ethical considerations, subjective judgment, and adaptability to new information as well as background information, play a pivotal role in decision-making in the medical domain. Hence, in this study, we prioritize a human-centered evaluation approach over objective metrics. To conduct this qualitative assessment, feedback was taken from three categories of dental professionals, both before and after the domain adaptation of the LLM. Each evaluator gave a score out of 10 for the output at report generation and Q&A in Sinhala. The rating process was iterated 10 times per professional to avoid biasing and random noise. The final score was determined by the weighted average of their ratings. The evaluators consisted of (1) a senior doctor with over 10 years of experience, who has completed a four-year residency training program and is a specialist in dental and maxillofacial radiology; (2) a doctor with 5 years of clinical experience; and (3) a graduate doctor--a recently qualified BDS degree holder with 1 year of experience. The weights for each category, tabulated in Table 8, were allocated by considering their professional experience. One of the bottlenecks faced when performing the quantitative evaluation of the Sinhala language discussion platform is the unavailability of an established benchmark dataset.