## 3 Results
### Evaluation of LLMs on USMLE Benchmarks  
To evaluate the reasoning capabilities of LLMs, a comprehensive benchmark was designed based on USMLE Step 1, Step 2, and Step 3-style multiple-choice questions. A total of 750 curated questions were used in the evaluation, equally distributed across the three steps. The focus was to assess the capability of base and fine-tuned LLMs to perform domain-specific reasoning in a clinical context. In this study, multiple configurations of Llama 3.1-based models were evaluated, specifically the 70B and 8B variants in both base (pretrained-only) and fine-tuned forms. Fine-tuning was conducted on a domain-specific corpus comprising 898,199 medical instances sourced from nine publicly available datasets covering clinical, consumer health, and biomedical content. Table 4 summarizes the data composition, including the number of samples per dataset, license type, and HuggingFace repository links. All sources are openly licensed and accessible for research use. Each model variant name encodes its key characteristics:  
* \"noquant\" indicates the model was not quantized, retaining full precision. * Numerical suffixes (e.g., 00, 01, 04, 07) represent the sampling temperature used during inference:
* 00 = deterministic output (temperature = 0.0)
* 01 = low variation (temperature = 0.1)
* 04 = moderate variation (temperature = 0.4)
* 07 = high variability (temperature = 0.7)
* \"dosample\" specifies diverse sampling strategies beyond temperature setting. **Fine-Tuning Setup**  
Fine-tuning was conducted using the Axolotl framework with 4-bit QLoRA for memory efficiency. Key hyperparameters included the following:  
* Epochs: 1-3
* Learning rate: 2 \(\times\) 10\({}^{-4}\)
* Micro-batch size: 2
* LoRA rank: 32
* LoRA alpha: 16
* Dropout: 0.05  
The dataset adhered to the Alpaca format, enhancing compatibility with instruction-tuned models. **Evaluation Metrics**  
Performance was measured in terms of accuracy on each USMLE step and overall average:  
* Step 1: Foundational knowledge
* Step 2: Clinical reasoning
* Step 3: Management and advanced diagnostics  
**Model Performance Comparison**  
Table 5 benchmarks our fine-tuned Llama-3.1 70B models against human examinees and cutting-edge GPT baselines. Accuracy is reported as the percentage of correct answers on the publicly available USMLE-style question set. The prompt template is given as follows: \begin{table}
\begin{tabular}{l l l l} \hline \hline
**Name** & \multicolumn{1}{c}{**Source Repository**} & **Instance Count** & **License** \\ \hline CORD-19 & medalpaca/medical\_meadow\_crd19 & 821,007 & CC BY-4.0 \\ \hline Medical Flashcards & medalpaca/medical\_meadow\_medical\_flashcards & 33,955 & CC0 \\ \hline MedGA & medalpaca/medical\_meadow\_medga & 10,187 & GPL-3.0 \\ \hline WikiDoc & medalpaca/medical\_meadow\_wikidoc & 10,000 & CC BY-SA-3.0 \\ \hline Health Advice & medalpaca/medical\_meadow\_health\_advice & 8676 & CC BY-4.0 \\ \hline WikiDoc—Patient Info & medalpaca/medical\_meadow\_wikidoc\_patient\_information & 5942 & CC BY-SA-3.0 \\ \hline MMMLU (medical) & medalpaca/medical\_meadow\_mmmlu & 3787 & MIT \\ \hline PubMed Causal & medalpaca/medical\_meadow\_pubmed\_causal & 2446 & CC BY-4.0 \\ \hline MEDIQA & medalpaca/medical\_meadow\_mediaq & 2208 & CC BY-NC-SA-4.0 \\ \hline Total & — & 898,199 & — \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Composition and licensing of the domain-specific corpus used for fine-tuning Llama 3.1 models. **Prompt Template Used for LLM Inference of USMLE Dataset**  
You are a helpful medical assistant. For each question, select the most appropriate answer from the provided options. The answer should be one of the letters (A, B, C, etc.). Using two closely related but independently tuned models allows us to isolate the effect of instruction-tuning recipe (NVIDIA vs. Meta) while holding core architecture constant. Parameter counts (8B vs. 70B) further reveal the scalability breakpoints highlighted in Tables 6-8.