# Multimodal AI and Large Language Models for Orthopantomography Radiology Report Generation and Q&A
## 3 Methodology
### LLM Model Selection for Report Generation and Domain Adaptation for LLM Training  
LLaMA 3 8B model was selected for radiology report generation and question-answering tasks based on its performance across multiple benchmark evaluations, including MMLU (73.0), IFEval (80.4), GPQA (32.8), and MGSM (68.9), as shown in Table 5. These benchmarks assess general knowledge, reasoning, and complex problem-solving abilities, which are crucial for generating clinically accurate and contextually coherent radiology reports. In comparison, models like Mistral 7B and Gemma 2 9B showed notably lower scores, reflecting limitations in complex text generation tasks. Additionally, encoder-only models such as BERT and RoBERTa are primarily designed for natural language understanding (NLU) tasks like classification and named entity recognition, and encoder-decoder models like T5 are better suited for sequence-to-sequence tasks. Neither architecture excels in long-form, free-text generation needed for radiology reports, further justifying our preference for decoder-only LLMs like LLaMA 3 8B in this domain. However, while large language models like LLaMA 3 8B have demonstrated remarkable capabilities in various natural language processing (NLP) tasks, their performance significantly degrades when applied to domains that differ from the training data. This is because LLMs are trained on massive datasets of text and code, which may not adequately represent the specific terminology, nuances, and context of a particular domain. To address this challenge, domain adaptation techniques are employed. In domain adaptation, an LLM already trained on a different domain is fine-tuned on a target domain using target domain-specific data while incorporating domain-specific constraints into the models' architecture, with the scope of improving accuracy, robustness, and generalization. LLM domain adaptation is particularly crucial in domains where data are scarce or sensitive, especially in the case of the medical domain. Through the adaptation of LLM to the medical domain, organizations  
Figure 4: Proposed Blip-2 architecture.  
can utilize its capabilities for tasks such as medical diagnosis assistance. Most importantly, this adaptation prioritized the privacy and confidentiality of sensitive medical information by allowing users to run a privately hosted version of an LLM model instead of relying on commercially available LLM models like ChatGPT, Gemini, etc. Hence, with domain adaptation, it is possible to adapt data-hungry LLMs to tasks in the data-scarce medical domain, enhancing their performance and applicability in real-world scenarios.  
Our study explored domain adaptation in LLMs through instruct fine-tuning and continued pre-training approaches. Although these approaches share some similarities, they also exhibit notable differences. The process of continued pre-training involves further training a pre-trained model using a large dataset, maintaining the same objective function and training setup as the initial pre-training phase. This methodology ensures the model's understanding of the domain's linguistic nuances and patterns while reinforcing its foundational language understanding. On the other hand, instruct fine-tuning generally involves adding a task-specific layer to the pre-trained model, utilizing a smaller, task-specific dataset for training. This process aims to align the model with a task-specific objective function to optimize the model for a particular task. In summary, these two fine-tuning approaches offer an effective mechanism for training a new LLM from the ground up, without demanding intensive resources using the domain adaptation concepts. Furthermore, we adopted domain adaptation in LLMs in three distinct instances within the scope of the study, as discussed below.