Evaluation of LLMs in Medical Text Summarization: The Role of Vocabulary Adaptation in High OOV Settings
Authors: Gunjan Balde, Soumyadeep Roy, Mainack Mondal, Niloy Ganguly
Accepted at: acl25

Evaluation of LLMs in Medical Text Summarization: The Role of Vocabulary Adaptation in High OOV Settings

Gunjan Balde*, Soumyadeep Roy, Mainack Mondal and Niloy Ganguly
Indian Institute of Technology Kharagpur
balde.gunjan0812@kgpian.iitkgp.ac.in
soumyadeep.roy9@iitkgp.ac.in
{mainack,niloy}@cse.iitkgp.ac.in

Abstract
Large Language Models (LLMs) recently achieved great success in medical text summarization by simply using in-context learning. However, these recent efforts do not perform fine-grained evaluations under difficult settings where LLMs might fail. They typically report performance scores over the entire dataset. Through our benchmarking study, we show that LLMs show a significant performance drop for data points with high concentration of out-of-vocabulary (OOV) words or with high novelty. Vocabulary adaptation is an intuitive solution to this vocabulary mismatch issue where the LLM vocabulary gets updated with certain expert domain (here, medical) words or subwords. An interesting finding from our study is that Llama-3.1, even with a vocabulary size of around 128K tokens, still faces over-fragmentation issue with medical words. To that end, we show vocabulary adaptation helps improve the LLM summarization performance even in difficult settings. Through extensive experimentation of multiple vocabulary adaptation strategies, two continual pretraining strategies, and three benchmark medical summarization datasets, we gain valuable insights into the role of vocabulary adaptation strategies for customizing LLMs to the medical domain. We also performed a human evaluation study with medical experts where they found that vocabulary adaptation results in more relevant and faithful summaries. Our codebase is made publicly available at https://github.com/gb-kgp/LLM-MedicalSummarization-Benchmark.

1 Introduction
Recent works like Clinsumm (Van Veen et al., 2024) explore various strategies to adapt LLMs in the task of medical text summarization. These strategies are (i) in-context learning (Brown et al., 2020; Lampinen et al., 2022) where a fixed number of exemplars is added in the prompt at inference time only; and (ii) parameter-efficient finetuning using QLoRA (Dettmers et al., 2024) because LLMs have billions of parameters which make complete finetuning computationally infeasible. However, these studies do not perform fine-grained evaluations considering challenging generation scenarios and typically report performance scores over the entire dataset. In this paper, we consider two such challenging scenarios primarily related to the high vocabulary mismatch between LLM vocabulary and the medical domain. Figure 1 provides an overview of our benchmarking setup.

![A diagram showing the benchmarking setup for fine-grained evaluation of LLMs in high OOV and high novelty setting for medical text summarization. Includes models, methods, datasets, vocabulary adaptation strategies, continual pretraining strategy, and evaluation subsets.](page_670_355_749_496.png)

Figure 1: We present the benchmarking setup for our fine-grained evaluation of LLMs in high OOV and high novelty setting for medical text summarization. Our benchmarking setup: (a) tests four recent mainstream SoTA models; (b) on three vocabulary adaptation methods; (c) using three query-focused medical summarization dataset; (d) over two continual pretraining procedures; (e) evaluated in five fine-grained scenarios based on OOV (Out-Of-Vocabulary) and Novel unigram concentration in SD (Source Document) and RS (Reference Summary) respectively. So, in total we evaluated \( 4 \times 3 \times 4 \times 2 \times 5 = 480 \) combinations.
We note that LLMs during tokenization typically over-fragment (important) medical words, resulting in higher OOV concentration (proportion of words that are split into more than one token by an LLM tokenizer) compared to the generic open-domain text (e.g., news domain). In this work, we focus on the ‘BASE’ model variants of four open-source, open-domain LLMs called Llama-2 (Touvron et al., 2023), Mistral (Jiang et al., 2023), Qwen2 (Yang et al., 2024) and Llama-3.1 (Dubey et al., 2024). Llama-3.1 and Qwen2 has a much larger vocabulary size of 128K and 151K tokens, which is significantly larger than Llama-2 and Mistral (almost four times), which have a vocabulary size of around 32K. Therefore, it would be interesting to investigate whether vocabulary adaptation still helps to improve Llama-3.1 and Qwen2. We observe in Table 1, that there is an increase of 17.89%, 20.25%, 14.06%, and 13.08% in fragment score (Rust et al., 2021) when comparing tokenization of words from the open domain with the medical domain using Llama-2, Mistral, Qwen2, and Llama-3.1 tokenizers. This over-fragmentation affects the encoding stage as the semantic meaning is lost due to poor tokenization (Hofmann et al., 2022) and during generation, the model has to generate more subwords to generate a medical word. Examples of words that are over-fragmented is shown in Table 2. Additionally, summarization datasets in the medical domain being from a specialized domain contain words in reference summaries (like disease names) which do not occur directly in the source document but require domain-knowledge to be inferred from the source document (novel words).

Vocabulary expansion is a potential solution for such domain adaptation challenges that primarily arise due to vocabulary mismatch between the LLM vocabulary and the target expert-domain tasks. In this process, the LLM’s vocabulary is expanded by incorporating tokens from the target domain that are not originally included in the LLM vocabulary. It has shown good performance for encoder-only models like BERT, RoBERTa on classification tasks (Hong et al., 2021; Hofmann et al., 2022; Lamproudis et al., 2022; Xu et al., 2023) and encoder-decoder models like BART, PEGASUS, and Transformers-Large on summarization and machine translation tasks (Xu et al., 2021; Nag et al., 2023; Balde et al., 2024b; Nag et al., 2024). Recent works are exploring vocabulary adaptation strategies for LLMs and proves useful for multilingual use-cases on models like Llama such as (Liu et al., 2023; Cui et al., 2024b,a; Liu et al., 2024; Gao et al., 2024; Nag et al., 2024; Tejaswi et al., 2024; Yamaguchi et al., 2024). However, these studies primarily focus on languages other than English, which are relatively underrepresented in the pretraining corpus of LLMs.

We make the following contributions:

• We investigate vocabulary adaptation strategies in using LLMs on tasks from the expert (medical) domain. Instead of evaluating a multilingual setting, we address a vocabulary mismatch within the same language (English).

• We perform a benchmarking study on medical text summarization with various vocabulary adaptation strategies tailored for LLMs. We perform fine-grained evaluation of LLMs in high OOV and high novelty settings (Table 6).

• We conduct a human evaluation study with experts who also found that vocabulary adaptation produces more relevant, coherent, and faithful summaries (Figure 3).

<table>
  <tr>
    <th></th>
    <th>CNN-DM</th>
    <th>PAC</th>
  </tr>
  <tr>
    <th colspan="3">Llama-2</th>
  </tr>
  <tr>
    <td>Fragment Score</td>
    <td>1.23</td>
    <td><b>1.45</b></td>
  </tr>
  <tr>
    <td>Split<sub>&gt; 3</sub></td>
    <td>21%</td>
    <td><b>35%</b></td>
  </tr>
  <tr>
    <th colspan="3">Mistral</th>
  </tr>
  <tr>
    <td>Fragment Score</td>
    <td>1.17</td>
    <td><b>1.41</b></td>
  </tr>
  <tr>
    <td>Split<sub>&gt; 3</sub></td>
    <td>21%</td>
    <td><b>34%</b></td>
  </tr>
  <tr>
    <th colspan="3">Llama-3.1</th>
  </tr>
  <tr>
    <td>Fragment Score</td>
    <td>1.07</td>
    <td><b>1.21</b></td>
  </tr>
  <tr>
    <td>Split<sub>&gt; 3</sub></td>
    <td>12%</td>
    <td><b>25%</b></td>
  </tr>
  <tr>
    <th colspan="3">Qwen2</th>
  </tr>
  <tr>
    <td>Fragment Score</td>
    <td>1.28</td>
    <td><b>1.46</b></td>
  </tr>
  <tr>
    <td>Split<sub>&gt; 3</sub></td>
    <td>17%</td>
    <td><b>28%</b></td>
  </tr>
</table>

Table 1: OOV concentration and fragment score observed for general domain dataset (CNN-DailyMail) and medical domain dataset (PAC) obtained using tokenizers of models: Llama-2 and Mistral (Vocabulary size: 32K), and Llama-3.1 and Qwen2 (Vocabulary size: 128K and 151K). Fragment score (Rust et al., 2021) is the average number of subwords a word is tokenized into; and Split<sub>&gt; 3</sub> is the fraction of words split more than thrice. We note that for all the model tokenizers, irrespective of the vocabulary size, medical domain words are over-fragmented leading to higher fragment score (highlighted in bold), compared to general domain.

2 Benchmarking Vocabulary Adaptation Strategies for LLMs

A vocabulary adaptation technique consists of three steps: (i) generating candidate vocabulary tokens
<table>
  <tr>
    <th>Medical Term</th>
    <th>Llama-2</th>
    <th>Llama-3.1</th>
  </tr>
  <tr>
    <th colspan="3">High OOV</th>
  </tr>
  <tr>
    <td>cardiomyopathy</td>
    <td>'_card', 'iom', 'y', 'op', 'ath', 'y'</td>
    <td>'card', 'i', 'omy', 'opathy'</td>
  </tr>
  <tr>
    <td>antipyretics</td>
    <td>'_ant', 'ip', 'y', 'ret', 'ics'</td>
    <td>'ant', 'ipy', 'ret', 'ics'</td>
  </tr>
  <tr>
    <th colspan="3">High novelty</th>
  </tr>
  <tr>
    <td>corticosteroid</td>
    <td>'_cort', 'ic', 'ost', 'ero', 'id'</td>
    <td>'c', 'ortic', 'oster', 'oid'</td>
  </tr>
  <tr>
    <td>antidepressant</td>
    <td>'_ant', 'ide', 'press', 'ants'</td>
    <td>'ant', 'idepress', 'ants'</td>
  </tr>
</table>

Table 2: Medical terms from reference summary of PubMedQA dataset with high OOV concentration (\( \text{Difficult}_{RS} \)) and high novelty (\( \text{Novel}_{RS} \)).

from target downstream datasets, (ii) selecting important vocabulary tokens from candidate set using some utility function (e.g., fragment score (Hong et al., 2021), corpus entropy (Xu et al., 2021)) to form added vocabulary, and (iii) learning the embeddings of the added vocabulary and integration into LLM. In this work, we benchmark the effect of all the three steps, combining the effect of first two steps: constructing the vocabulary and adding candidate tokens from the target domain to finalize the vocabulary for integration into the LLM as one, and then checking effect of last step: on how to train the embeddings.

2.1 Vocabulary Adaptation Methods

We consider two types of datasets while constructing the vocabularies to be added to the PLM vocabulary: (i) PubMed Abstract Collection (PAC), a collection of around 300K PubMed abstracts used for intermediate fine-tuning by MEDVOC (Balde et al., 2024b), and (ii) Target Task (TGT Task), the target downstream task dataset for which the vocabulary is to be constructed. We now describe the different vocabulary adaptation methods used for our benchmarking study.

MEDVOC. MEDVOC (Balde et al., 2024b) is a SoTA vocabulary adaptation strategy for adapting PLMs like BART and PEGASUS, on medical summarization tasks. First, candidate vocabularies are constructed on the medical OOV words from a domain-specific corpus (PAC) – \( V_{PAC} \), and a downstream task dataset – \( V_{TGT} \). Then, an optimal vocabulary (\( V_{MEDVOC} \)), that lies at the intersection of \( V_{PAC} \) and \( V_{TGT} \) is chosen via a hyperparameter search. The utility function for the hyperparameter search is fragment score (Rust et al., 2021), defined as the average number of subwords a word from a corpus \( C \) is tokenized into by a tokenizer using vocabulary \( \mathcal{V} \). The vocabulary configuration within the neighborhood of the optimal vocabulary is finally chosen to avoid overfitting on large vocabulary sizes.

MEDVOC-LLM. This is a variant of MEDVOC adapted for LLM vocabularies and tokenizers. We identify two key issues in MEDVOC: (i) many of the vocabulary terms added directly from \( V_{PAC} \), did not occur even once in the reference summaries of the train set of downstream target task – their addition did not contribute during generation, and (ii) certain added vocabulary terms were a mixture of numerals and punctuations (e.g., -9,) – which is not consistent with the tokenization scheme for LLMs considered in this study\(^1\) (Llama-2 and Llama-3.1). Thus, in this approach, we clean the vocabularies generated by MEDVOC by removing the terms from both categories.

Overhead in Previous Vocabulary Adaptation Strategy. Consider the word ‘cholesterol’, which is not present in the Llama-3.1 model’s vocabulary and is tokenized by the Llama-3.1 tokenizer as [chole, le, sterol]. Since a merge rule operates on pairs of tokens, we need to iteratively add pairs from left to right, as shown in Table 3.

<table>
  <tr>
    <th>Token</th>
    <th>Merge Rule</th>
  </tr>
  <tr>
    <td>‘chole’</td>
    <td>[ch, ole]</td>
  </tr>
  <tr>
    <td>‘cholesterol’</td>
    <td>[chole, sterol]</td>
  </tr>
</table>

Table 3: Illustration of iterative addition of tokens to add a target token ‘cholesterol’ in the vocabulary.

Thus, in order to add ‘cholesterol’ to the vocabulary, we need to add one extra token [chole]. Specifically, for Llama-3.1, we observe an overall addition of 20% of extra such tokens, which are seldom used once the complete word is added into the vocabulary. This can lead to reduced performance in downstream tasks. To that end, we present ScafFix to address this issue with existing vocabulary adaptation strategies.

ScafFix. Unlike the vocabulary adaptation strategies described till now, ScafFix constructs the candidate set for added vocabulary by directly considering the medical words and ignores the tok—

\(^1\)Llama tokenizers explicitly set apart digits as individual tokens (Touvron et al., 2023)
enization step for forming candidate subwords. We directly select \( x \) tokens (where \( x \) represents the quota, set to 500 in this case, in steps of 50) from the candidate vocabulary tokens, ranking them by their frequency in descending order. We then follow the MEDVOC-based hyperparameter search optimizing fragment score to obtain the optimal vocabulary to be added. To offset the absence of such derivative tokens, we use AdaptBPE tokenization scheme (Balde et al., 2024a) instead of the standard model tokenizers. Instead of directly utilizing merge rules, AdaptBPE first checks whether a part of the input token (using longest-first match) directly exists in the added vocabulary, preserves it from splitting, and then runs the merge-based byte-pair encoding scheme iteratively on the rest of the input. Here, we avoid including the scaffolding tokens (Cognetta et al., 2024; Bauwens and Delobelle, 2024; Chizhov et al., 2024; Lian et al., 2025) during the vocabulary addition phase; these are derivative tokens that remain under-trained once the whole word is added to the vocabulary.

2.2 Learning the Added Vocabulary Embeddings using Continued Pretraining

The embedding of the newly added vocabulary is initialized as the average of the embeddings of the existing subwords in the vocabulary (Yamaguchi et al., 2024). We explore two continual pretraining (Gururangan et al., 2020; Tejaswi et al., 2024; Yamaguchi et al., 2024) strategies to train these embeddings on target domain text (20K random documents from the PubMed Abstract Collection, in our case). Continual pretraining uses the same training objective of ‘Next Token Prediction’ as the autoregressive language modeling objective, and optimizes the standard negative log-likelihood loss. We use the popular parameter-efficient fine-tuning technique known as ‘Low-Rank Adaptors’ (LoRA) (Mangrulkar et al., 2022) because end-to-end training of LLMs is computationally infeasible. We explore two continual pretraining strategies:

• End-To-End: The model is trained in an end-to-end manner by freezing all the base model layers except the input and output embedding layers and training LoRA adapters.

• Two-Stage: First, the entire model is frozen along with the LoRA layers except for the input and output embedding layers for a short duration, then unfreeze the LoRA adapters and train the LoRA adapters along with the embedding layers (Cui et al., 2024b; Yamaguchi et al., 2024). The second approach leads to more stable training and avoids overfitting to the initial LLM embedding space.

<table>
  <tr>
    <th>Category</th>
    <th>Description</th>
  </tr>
  <tr>
    <th colspan="2">Categories of Words</th>
  </tr>
  <tr>
    <td>Difficult-OOV</td>
    <td>Medical words that are split more than thrice by model tokenizers</td>
  </tr>
  <tr>
    <td>Novel</td>
    <td>Words in the summary that are not present in the source document</td>
  </tr>
  <tr>
    <td>All-OOV</td>
    <td>Medical words that are split more than once by model tokenizers</td>
  </tr>
  <tr>
    <th colspan="2">Evaluation Setting (Top ten percentile)</th>
  </tr>
  <tr>
    <td>Difficult<sub>RS</sub></td>
    <td>High Difficult-OOV concentration in the reference summaries</td>
  </tr>
  <tr>
    <td>Difficult<sub>SD</sub></td>
    <td>High Difficult-OOV concentration in the source document</td>
  </tr>
  <tr>
    <td>Novel<sub>RS</sub></td>
    <td>High Novel concentration in the reference summaries</td>
  </tr>
  <tr>
    <td>All<sub>SD</sub></td>
    <td>High All-OOV concentration in the source document</td>
  </tr>
  <tr>
    <td>All<sub>RS</sub></td>
    <td>High All-OOV concentration in the reference summaries</td>
  </tr>
</table>

Table 4: Challenging fine-grained evaluation scenarios considered in this study. We focus on the subset that contains high OOV concentration and novelty

3 Experimental Setup

We use in-context learning (ICL) (Brown et al., 2020; Van Veen et al., 2024) along with greedy decoding for generating summaries from LLMs. We follow the approach similar to ClinSumm (Van Veen et al., 2024), where examples are sampled from the train set using the similarity computed using PubMedBERT\(^2\) with the given test data point. The template for prompting is shown in Appendix A (Table 9). We also describe the fine-grained evaluation setup used for this benchmarking study in Table 4.

Next, we describe the benchmark medical text summarization datasets used, followed by details on evaluation metrics, baseline models, and training details.

3.1 Datasets

In this work, we focus on three query-focused medical summarization datasets such as BioASQ (Tstsarasonis et al., 2015), EBM (Mollá and Santiago-Martínez, 2011), and PubMedQA (Jin et al., 2019).

\footnotetext{2https://huggingface.co/pritamdeka/PubMedBERT-mnli-snli-scinli-scitail-mednli-stsb}
A data point consists of a Query (Q) and a context (PubMed Abstract) as the input to the model. The gold-standard reference summary (RS) summarizes the abstract based on the query. We present the overall dataset characteristics in Table 5.

PubMedQA. PubMedQA is a question-answering dataset with 1000 human-annotated data points. Here, input is a query appended to a PubMed abstract, which forms the source document. We consider the ‘long answer’ as the reference summaries.

EBM. In this single-document summarization task, the input comprises a query paired with a PubMed abstract. The task is to generate a concise summary that addresses the query, using the provided abstract as its context.

BioASQ. We use the Phase-B query-focused summarization task of BioASQ-9B. The input includes a question along with relevant PubMed abstracts. For the summarization task, we use the ideal answer as the reference summary. Here, we explore two variants of a source document — (i) Snippets (BioASQ-S): the question followed by the list of relevant snippets from a collection of PubMed Abstracts, in line with the MEDVOC paper (Balde et al., 2024b), and (ii) BioASQ-Main Abstract (BioASQ-M): the question followed by the complete Pubmed abstracts.

3.2 Evaluation Metrics
We evaluate the model-generated summaries using Rouge-L (R-L) to measure informativeness and coherence, and Concept-Score (CSr) to measure faithfulness (Zhang et al., 2023). Concept-Score measures the overlap of UMLS medical concepts (computed using QuickUMLS tool (Soldaini and Goharian, 2016)) between the generated and reference summaries. We use Rouge-L as the primary comparison metric, in line with prior studies (Fabri et al., 2021; Yuan et al., 2022; Balde et al., 2024a,b).

3.3 Baseline Models
The following baseline models do not update the LLM vocabulary:

• BASE. ‘BASE’ corresponds to the base variant of original LLM without any vocabulary adaptation and continual pretraining. We consider 7B variant of Llama-2 (Model id: meta-llama/Llama-2-7b-hf), Mistral (Model id: mistralai/Mistral-7B-v0.1), Qwen2 (Model id: Qwen/Qwen2-7B), and 8B variant of Llama-3.1 (Model id: meta-llama/Llama-3.1-8B) as our BASE models.

• Continual Pretraining (CPT-Only). CPT-Only corresponds to the original BASE LLM that has undergone the continual pretraining (CPT) but without any vocabulary adaptation. It serves as a strong baseline when comparing against vocabulary-adapted models.

3.4 Implementation Details
We provide basic details of implementation such as details on LoRA, size of pretraining corpus, and training hyperparameters.

Continual Pretraining using LoRA. We use one A100 40 GB GPU to carry out the pretraining. We use LoRA (Hu et al., 2022; Mangrulkar et al., 2022) to carry out the pretraining in this resource-constrained setting. The LoRA adapters are applied to all the linear modules in the model. These include \( \{k_{proj}, q_{proj}, v_{proj}, o_{proj}\} \) modules from self attention layers along with \( \{gate_{proj}, up_{proj}, and\ down_{proj}\} \) modules from MLP layers. We use a consistent LoRA configuration of a rank value of 32 and an alpha value of 64 for fair comparison.

Size of Pretraining Corpus. As continual pretraining with all the 312K documents from the PubMed Abstracts Collection (PAC) is computationally infeasible for LLMs, we perform a hyperparameter search to identify an optimal dataset size. We experiment with various dataset sizes of 10K, 20K, 50K, and 100K, and observe that the performance observed for 20K which only takes 6 hours of pretraining was comparable with the of 100K setting, which took approximately 40 hrs of pretraining. Therefore, we continually pretrain over the BASE model with randomly selected 20K documents from PAC.

Training Hyperparameters. We use a global batch size of 32 (on device: 8 with gradient accumulation: 4), and a learning rate of \( 1e^{-4} \). In End-to-End pretraining procedure, LoRA layers of the model were trained end-to-end for 5 epochs. In case of Two-Stage pretraining procedure, the embedding layers training phase and keeping LoRA layers frozen, was carried out for 2 epochs on 10K PAC samples. Then both the LoRA and embedding layers were trained end-to-end on 20K PAC
<table>
  <tr>
    <th rowspan="2">Dataset</th>
    <th rowspan="2">Test Set Size</th>
    <th colspan="2">Token Count of Reference Summaries</th>
    <th colspan="4">OOV Concentration Split more than once (in %)</th>
    <th colspan="4">OOV Concentration Split more than thrice (in %)</th>
    <th rowspan="2">Unigram Novelty (in %)</th>
  </tr>
  <tr>
    <th>Llama-2</th>
    <th>Llama-3.1</th>
    <th>SD</th>
    <th>RS</th>
    <th>Llama-2</th>
    <th>Llama-3.1</th>
    <th>SD</th>
    <th>RS</th>
    <th>Llama-2</th>
    <th>Llama-3.1</th>
    <th>SD</th>
    <th>RS</th>
  </tr>
  <tr>
    <td>PubMedQA</td>
    <td>500</td>
    <td>63</td>
    <td></td>
    <td>25</td>
    <td>36.67</td>
    <td>38.00</td>
    <td>43.68</td>
    <td>45.65</td>
    <td>4.91</td>
    <td>4.65</td>
    <td>2.61</td>
    <td>2.42</td>
    <td>41.32</td>
  </tr>
  <tr>
    <td>EBM</td>
    <td>424</td>
    <td>112</td>
    <td></td>
    <td>91</td>
    <td>38.97</td>
    <td>40.90</td>
    <td>45.60</td>
    <td>46.23</td>
    <td>6.65</td>
    <td>7.92</td>
    <td>3.90</td>
    <td>5.17</td>
    <td>47.15</td>
  </tr>
  <tr>
    <td>BioASQ-M</td>
    <td>963</td>
    <td>85</td>
    <td></td>
    <td>69</td>
    <td>46.20</td>
    <td>50.64</td>
    <td>52.03</td>
    <td>56.61</td>
    <td>9.12</td>
    <td>11.04</td>
    <td>5.55</td>
    <td>7.09</td>
    <td>42.58</td>
  </tr>
  <tr>
    <td>BioASQ-S</td>
    <td>496</td>
    <td>73</td>
    <td></td>
    <td>58</td>
    <td>47.12</td>
    <td>50.00</td>
    <td>52.00</td>
    <td>57.15</td>
    <td>8.70</td>
    <td>9.10</td>
    <td>4.76</td>
    <td>4.55</td>
    <td>41.11</td>
  </tr>
</table>

Table 5: Medical text summarization datasets used for evaluation. We have three key observations: (i) BioASQ has the highest OOV concentration; (ii) EBM has highest novelty concentration; and (iii) EBM has the longest length reference summaries.

samples for 3 epochs. In both the pretraining procedures, the base layers of the models were kept frozen throughout the training process.

4 Experimental Results

Table 6 shows the performance comparison of the different vocabulary adaptation strategies on Llama-2 and Llama-3.1 models. We report the best performance among the two continual pretraining variants of 'End-to-End' and 'Two-Stage' as previously described in Section 2.2. The performance values of the individual settings is added to the Appendix B. Since Mistral vocabulary size is same as Llama-2 (32K), and Qwen2 vocabulary size is similar as Llama-3.1 (151K and 128K), we add the Mistral (Table 13) and Qwen2 (Table 14) results to the Appendix B. We summarize the key results for Qwen2 and Mistral model in RQ7. We observe that vocabulary adaptation leads to performance improvement in terms of Rouge-L for Llama-2 and Llama-3.1 in seven out of eight settings (except for Llama-2 on EBM dataset). In terms of Concept-Score which is a proxy measure for faithfulness (Zhang et al., 2023), Figure 2 shows that at least one vocabulary adaptation performs the best in five out of six settings, except for Llama-3.1 on PubMedQA dataset.

4.1 Discussion of Results for Vocabulary Adaptation with LLMs

We focus on the fine-grained analysis of LLMs in high OOV and high novelty data points. The Rouge-LCS values observed for each such category are reported in Table 6. In total, we benchmark an LLM on 12 settings: 8 OOV-related settings and 4 novelty-related settings. Here we report the best performing training method in DifficultSD, DifficultNS and Novelty setting. The complete results can be found in Table 11 (for Llama-2) and Table 12 (for Llama-3.1). The Concept score values are shown in Figure 2. We find that at least one vocabulary adaptation strategy improves over BASE in a total of five out of six settings (LLM-Dataset pairs). The overall average improvement for 18.75% for Llama-2 and 14.82% for Llama-3.1 models across datasets. We present two representative examples from EBM and PubMedQA datasets using Llama-2 models in Appendix D.

Now, we explore research questions to evaluate LLMs in difficult settings and scenarios where vocabulary adaptation strategies does not help.

RQ1: Vocabulary adaptation outperform BASE model on full test data (TestFull). We note that at least one of the vocabulary adaptation strategies (best of MEDVOC-LLM and ScafFix) improves over BASE in 5 out of 8 settings. The average performance improvement is 3.68% for Llama-2 and 4.57% for Llama-3.1. The methods where we see improvements are: MEDVOC-LLM (2/5) and ScafFix (3/5) settings. Thus on the entire test set, ScafFix, is the best-performing vocabulary adaptation method. It even outperforms CPT-Only in six out of eight settings by a margin of 2.97%.

RQ2: CPT-Only improves over BASE in high novelty and OOV concentration. We find that CPT-Only (Continual Pretraining without any vocabulary adaptation) improves over BASE in 4 out of 8 (Llama-2: 2; Llama-3.1: 2) high novelty settings. CPT-Only improves over BASE in 11 out of 16 (Llama-2: 6; Llama-3.1: 5) higher OOV settings. However, we observe that at least one vocabulary adaptation strategy improves over CPT-Only in high OOV (13 out of 16 settings) and high novelty (in all 8 setting); thus necessitating the need for vocabulary adaptation.

RQ3: Vocabulary adaptation helps in high OOV concentrations in reference summaries and source documents. We observe that vocabulary adaptation (best of MEDVOC-LLM and ScafFix) outperforms BASE in 14 out of 16 set-
<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="4">Llama-2</th>
    <th colspan="4">Llama-3.1</th>
  </tr>
  <tr>
    <th>TestFull</th>
    <th>DifficultRS</th>
    <th>NovelRS</th>
    <th>DifficultSP</th>
    <th>Overall</th>
    <th>TestFull</th>
    <th>DifficultRS</th>
    <th>NovelRS</th>
    <th>DifficultSP</th>
    <th>Overall</th>
  </tr>
  <tr>
    <td>BASE</td>
    <td>26.33</td>
    <td>24.40</td>
    <td>19.78</td>
    <td>23.53</td>
    <td>23.51</td>
    <td><b>28.10</b></td>
    <td>26.87</td>
    <td>18.87</td>
    <td>24.46</td>
    <td>24.58</td>
  </tr>
  <tr>
    <td>CPT-only</td>
    <td>27.12</td>
    <td>28.00</td>
    <td>22.21</td>
    <td>25.97</td>
    <td>25.83</td>
    <td>26.62</td>
    <td><b>27.08</b></td>
    <td>21.07</td>
    <td>24.52</td>
    <td>24.82</td>
  </tr>
  <tr>
    <td>MEDVOC</td>
    <td>26.50</td>
    <td>22.88</td>
    <td>20.14</td>
    <td>24.35</td>
    <td>23.47</td>
    <td>27.86</td>
    <td>26.23</td>
    <td>21.87</td>
    <td>24.68</td>
    <td>25.16</td>
  </tr>
  <tr>
    <td>MEDVOC-LLM</td>
    <td>26.90</td>
    <td>26.67</td>
    <td>20.00</td>
    <td>24.49</td>
    <td>24.52</td>
    <td>27.69</td>
    <td>26.67</td>
    <td>21.74</td>
    <td><b>26.51</b></td>
    <td><b>25.65</b></td>
  </tr>
  <tr>
    <td>ScafFix</td>
    <td><b>27.61</b></td>
    <td><b>28.57</b></td>
    <td><b>22.22</b></td>
    <td><b>26.32</b></td>
    <td><b>26.18</b></td>
    <td>27.67</td>
    <td>25.00</td>
    <td><b>22.22</b></td>
    <td>23.85</td>
    <td>24.69</td>
  </tr>
</table>

<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="4">PubMedQA</th>
    <th colspan="4">EBM</th>
  </tr>
  <tr>
    <th>TestFull</th>
    <th>DifficultRS</th>
    <th>NovelRS</th>
    <th>DifficultSP</th>
    <th>Overall</th>
    <th>TestFull</th>
    <th>DifficultRS</th>
    <th>NovelRS</th>
    <th>DifficultSP</th>
    <th>Overall</th>
  </tr>
  <tr>
    <td>BASE</td>
    <td>18.56</td>
    <td><b>16.00</b></td>
    <td>11.20</td>
    <td>17.33</td>
    <td>15.77</td>
    <td>20.04</td>
    <td>14.54</td>
    <td>11.37</td>
    <td>15.79</td>
    <td>15.44</td>
  </tr>
  <tr>
    <td>CPT-only</td>
    <td>19.13</td>
    <td>15.87</td>
    <td>11.77</td>
    <td><b>19.05</b></td>
    <td><b>16.46</b></td>
    <td>20.13</td>
    <td>17.15</td>
    <td>13.33</td>
    <td>16.98</td>
    <td>16.90</td>
  </tr>
  <tr>
    <td>MEDVOC</td>
    <td>18.60</td>
    <td>15.38</td>
    <td>12.50</td>
    <td>18.65</td>
    <td>16.28</td>
    <td>20.31</td>
    <td>16.03</td>
    <td>12.17</td>
    <td>17.93</td>
    <td>16.61</td>
  </tr>
  <tr>
    <td>MEDVOC-LLM</td>
    <td><b>19.27</b></td>
    <td>14.71</td>
    <td><b>12.90</b></td>
    <td>17.54</td>
    <td>16.11</td>
    <td>20.75</td>
    <td>16.40</td>
    <td>13.04</td>
    <td><b>18.39</b></td>
    <td><b>17.15</b></td>
  </tr>
  <tr>
    <td>ScafFix</td>
    <td>18.65</td>
    <td>14.71</td>
    <td>11.77</td>
    <td>17.40</td>
    <td>15.63</td>
    <td><b>20.79</b></td>
    <td><b>17.17</b></td>
    <td><b>13.80</b></td>
    <td>16.18</td>
    <td>16.99</td>
  </tr>
</table>

<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="4">BioASQ-S</th>
    <th colspan="4">BioASQ-M</th>
  </tr>
  <tr>
    <th>TestFull</th>
    <th>DifficultRS</th>
    <th>NovelRS</th>
    <th>DifficultSP</th>
    <th>Overall</th>
    <th>TestFull</th>
    <th>DifficultRS</th>
    <th>NovelRS</th>
    <th>DifficultSP</th>
    <th>Overall</th>
  </tr>
  <tr>
    <td>BASE</td>
    <td>32.12</td>
    <td>26.12</td>
    <td>20.53</td>
    <td>35.72</td>
    <td>28.62</td>
    <td>35.25</td>
    <td>29.60</td>
    <td>18.18</td>
    <td>27.03</td>
    <td>27.52</td>
  </tr>
  <tr>
    <td>CPT-only</td>
    <td><b>33.30</b></td>
    <td>29.22</td>
    <td>20.00</td>
    <td>30.33</td>
    <td>28.21</td>
    <td>36.01</td>
    <td>24.70</td>
    <td>17.14</td>
    <td>29.41</td>
    <td>26.82</td>
  </tr>
  <tr>
    <td>MEDVOC</td>
    <td>32.26</td>
    <td>28.17</td>
    <td>18.18</td>
    <td>27.40</td>
    <td>26.50</td>
    <td>37.01</td>
    <td>29.79</td>
    <td>18.18</td>
    <td>32.43</td>
    <td>29.35</td>
  </tr>
  <tr>
    <td>MEDVOC-LLM</td>
    <td>32.40</td>
    <td><b>29.41</b></td>
    <td>20.29</td>
    <td>30.33</td>
    <td>28.11</td>
    <td><b>37.15</b></td>
    <td><b>32.89</b></td>
    <td>20.29</td>
    <td><b>37.84</b></td>
    <td><b>32.04</b></td>
  </tr>
  <tr>
    <td>ScafFix</td>
    <td>32.88</td>
    <td>29.29</td>
    <td><b>22.22</b></td>
    <td><b>36.44</b></td>
    <td><b>30.21</b></td>
    <td>36.70</td>
    <td>32.26</td>
    <td><b>20.90</b></td>
    <td>34.15</td>
    <td>31.00</td>
  </tr>
</table>

Table 6: Fine-grained performance evaluation of vocabulary adaptation strategies and baseline modes in terms of Rouge-L (R-L). The values represent the best among the two continual pretraining strategies of ‘End-to-End’ and ‘Two-Stage’. The “Overall” column represents the average value of TestFull, DifficultRS, NovelRS and DifficultSP. Vocabulary adaptation (best of MEDVOC-LLM and ScafFix) improves over BASE in 7 out of 8 overall settings.

tings. For these settings, the average performance improvement over BASE is 8.74% and 14.64% for Llama-2 and Llama-3.1 respectively. Thus, MEDVOC-LLM is the best vocabulary adaptation strategy in scenarios of high OOV concentration in reference summaries and source documents.

RQ4: Vocabulary adaptation helps in high novelty settings. We observe that vocabulary adaptation improves over BASE in 6 out of 8 high novelty settings. The average performance improvement (wherever observed) is 11.92% and 18.03% for Llama-2 and Llama-3.1 when compared to BASE. Thus, ScafFix is the best vocabulary adaptation strategy in high novelty settings as it outperformed in five out of eight settings.

<table>
  <tr>
    <th>Word</th>
    <th>BASE tokenization</th>
    <th>ScafFix tokenization</th>
  </tr>
  <tr>
    <td>microbiologically</td>
    <td>['Microbi', 'ologically']</td>
    <td>['Micro', 'erco', 'iological', 'ly']</td>
  </tr>
  <tr>
    <td>inhibitory</td>
    <td>['Ginhib', 'itory']</td>
    <td>['G', 'inhibitor', 'y']</td>
  </tr>
  <tr>
    <td>chronically</td>
    <td>['Gchron', 'ically']</td>
    <td>['G', 'chronic', 'ally']</td>
  </tr>
  <tr>
    <td>antibacterial</td>
    <td>['Gantib', 'acterial']</td>
    <td>['Ganti', 'bacteria', 'l']</td>
  </tr>
</table>

Table 7: Samples of words from PubMedQA corpus. For each word, we observe that ScafFix tokenization preserves the morphological boundary while tokenization, unlike BASE tokenization where subwords cross morphological boundaries (Bauwens and Delobelle, 2024).

RQ5: Comparison of added vocabulary sizes among vocabulary adaptation strategies. The details of the added vocabulary and fragment score for the different vocabulary adaptation strategies are provided in Table 8 (Appendix A). We observe that ScafFix reduces fragment score by 29.19% over BASE by adding the minimum amount of added vocabulary terms as compared to the MEDVOC and MEDVOC-LLM. The lowest vocabulary size due to removal of scaffolding tokens leads to a significant drop in the number of under-trained tokens in the LLM vocabulary. This makes the training phase less noisy and yields superior performance during inference.

RQ6: Vocabulary adaptation does not help much in case of extractive summaries, low OOV concentration and low novelty. We note that all the vocabulary adaptation techniques struggle to outperform both BASE and CPT-Only in PubMedQA (Llama-3.1), BioASQ-S (Llama-2), and BioASQ-M (Llama-2 and Llama-3.1). When compared in the entire test set; there is a performance drop of 1.39% across these two settings. This may be because PubMedQA and BioASQ-S have low novelty. Specifically, for BioASQ-S, the unigram novelty is just 4.11% (see Table 5), therefore, the high novelty threshold for top-ten percentile was 35%. Thus, we take the average of EBM and PubMedQA high novelty values of 60%. This results
Figure 2: Concept Score (CSr) observed for (a) PubMedQA; (b) EBM; and (c) BioASQ-Μ averaged over DifficultSD, DifficultSG, and NoveltyS setting. At least one vocabulary adaptation strategy improves over BASE in a total of 5 out of 6 comparisons. The overall average improvement for 18.75% for Llama-2 and 14.82% for Llama-3.1 models across datasets.

in vocabulary adaptation outperforming BASE in high novelty setting of BioASQ-S.

In case of BioASQ-Μ, we note that although it is a dataset with higher unigram novelty, the summaries where ScafFix fails have a higher Rouge-L overlap with the Source Document as compared to the summaries where ScafFix does better than BASE, thus making the inference easier irrespective of OOV and Novelty concentration. We provide a detailed error analysis observed in the performance gap in Appendix B (Table 10)

RQ7: Proposed vocabulary adaptation methods generalizes to other LLMs such as Qwen2 and Mistral. In case of Qwen2 (Table 14), we observe that at least one vocabulary adaptation strategies improve over BASE and CPT-Only in 7 out of 12 comparisons. In case of Mistral (Table 13), we observe that at least one of the vocabulary adaptation strategies improved over BASE and CPT-Only in one out of 12 comparisons. In case of both the models (details in Appendix B), we observe a similar trend with vocabulary adaptation not helping in case of low OOV and novelty concentration (specifically in BioASQ-S) as we observed in RQ6. These findings highlight that our results are potentially generalizable to multiple LLMs.

4.2 Human Evaluation

We randomly select 30 test data points uniformly across the three datasets and two models that have higher expert OOV concentration. We use the Prolific platform to recruit medical experts for annotating summary pairs of ScafFix model and BASE across three aspects (Fabbri et al., 2021; Zhang et al., 2023; Balde et al., 2024b) namely relevance, coherence (on a Likert scale of 1 to 5), and faithfulness (binary). Each annotator was given 30 minutes to evaluate 10 summaries and was compensated at a rate of 8 UK pounds per hour (see Appendix C for more details), and each summary par was evaluated by three annotators. Figure 3 shows the human evaluation results where the ScafFix method generates more faithful summaries (93.34% versus 83.34% of summaries are faithful), and more relevant summaries, where 93.34% of data points get a score \( \geq 4 \) in Likert scale, as compared to 70% by BASE.

Figure 3: Human evaluation scores in high OOV concentration on ‘End-to-End’ continual pretraining strategy. Vocab corresponds to ScafFix model which produces more relevant, coherent, and faithful summaries during human evaluation with medical experts.

5 Related Works

Vocabulary Expansion in LLMs. Recent research has focused on enhancing large language models (LLMs) through domain-specific vocabulary optimization. Liu et al. (2024) introduces VEGAD, an adaptive method for selecting an opti-
mal subset of domain vocabulary, which enhances performance on both specialized and general tasks, validated on Chinese datasets. Similarly, Gao et al. (2024) introduced VE-KD, a method combining vocabulary expansion and knowledge distillation to train efficient domain-specific language models for BioBERT and PubMedBERT on biomedical tasks. These approaches align with broader efforts, such as, Chatlaw (Cui et al., 2024a), an AI legal assistant, that employed a knowledge graph-enhanced mixture-of-experts model to address legal domain challenges, and (Cui et al., 2024b), who developed efficient text encoding strategies for Chinese LLMs. Similar efforts have been put in other languages (Nag et al., 2024; Tejaswi et al., 2024; Yamaguchi et al., 2024). However, none of the works benchmark the effect in the fine-grained manner that we aimed to show in this study.

Over-fragmentation in Medical domain. Over-fragmentation (splitting of domain words in more than one subword) in tokenization is a significant challenge in adapting large language models (LLMs) to the medical domain due to specialized terminology. Si et al. (2019) highlighted the limitations of traditional tokenization on the task of clinical concept extraction where they show models like BERT poorly tokenize medical named entities. Nguyen et al. (2019) emphasized the impact of lexical segmentation on transformer-based models and suggested that not only using a domain-specific vocabulary helps, but also a continual pretraining phase helps. Yuan et al. (2022) introduced BioBART and Labrak et al. (2024) introduced BioMistral, biomedical-specific generative model designed to address domain-specific tokenization challenges that outperformed various SoTA methods on different kinds of task like summarization and question-answering. Liu et al. (2023) proposed task-adaptive tokenization to enhance long-form text generation by dynamically adjusting tokenization for domain-specific semantics by going beyond word boundaries during tokenization. Our work builds on these advancements by benchmarking the effect of refining tokenization on recent Llama-2, Llama-3.1, Mistral, and Qwen2 models.

6 Conclusion

This work is a first step towards understanding how vocabulary adaptation strategies effect the performance of general-purpose LLMs in medical text summarization. We first show that general-purpose LLMs fail in certain challenging generation scenarios where reference summaries have high OOV concentration and high novelty. We then benchmark the performance of three vocabulary adaptation strategies on four models: Llama-2 7B, Mistral 7B, Qwen2 7B, and Llama-3.1 8B model; over three biomedical summarization datasets and two continual pretraining strategies. Llama-3.1 and Qwen2 even with a vocabulary size of 128K and 151K tokens, still faces the over-fragmentation issue with medical words, and vocabulary adaptation is shown to help improve the LLM summarization performance. Medical experts find that vocabulary adaptation improves the relevance, coherence, and faithfulness of medical summaries. We make the codebase available for reproducibility purposes3.

7 Limitations

We identify two main limitations of this work. Firstly, because of hardware constraints we could not explore much larger variants (like 13B and 70B) of the models considered in the study. Second, we note that all the results provided in this paper are generated by applying LoRA during the pretraining phase. This might make a difference when replicating the results with full-scale fine-tuning without any parameter efficient fine-tuning strategies.

8 Ethics Statement and Broader Impact

It is a well known fact that LLMs are often prone to hallucinations, producing outputs that could not be verified as-is from the source context without any domain-specific knowledge. While the proposed methods here do improve faithfulness, the quality of the summaries generated is still not ready for deployment to the public without evaluating the safety perspective of such responses. We believe more research is needed to align the outputs of these LLMs for such high-stake domains like healthcare where a single misinformation (or disinformation) could lead to drastic consequences.

Acknowledgments

We thank the Ministry of Education, Govt of India, for supporting Gunjan Balde with Prime Minister Research Fellowship during his Ph.D. tenure. This research was partially funded by a Google Academic Research Award.

3https://github.com/gb-kgp/LLM-MedicalSummarization-Benchmark
References

Gunjan Balde, Soumyadeep Roy, et al. 2024a. Adaptive BPE tokenization for enhanced vocabulary adaptation in finetuning pretrained language models. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 14724–14733, Miami, Florida, USA. Association for Computational Linguistics.

Gunjan Balde, Soumyadeep Roy, et al. 2024b. MED-VOC: Vocabulary adaptation for fine-tuning pretrained language models on medical text summarization. In Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence, IJCAI-24, pages 6180–6188. International Joint Conferences on Artificial Intelligence Organization. Main Track.

Thomas Bauwens and Pieter Delobelle. 2024. BPE-knockout: Pruning pre-existing BPE tokenisers with backwards-compatible morphological semi-supervision. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 5810–5832, Mexico City, Mexico. Association for Computational Linguistics.

Tom Brown, Benjamin Mann, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901.

Pavel Chizhov, Catherine Arnett, et al. 2024. BPE gets picky: Efficient vocabulary refinement during tokenizer training. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 16587–16604, Miami, Florida, USA. Association for Computational Linguistics.

Marco Cognetta, Tatsuya Hiraoka, et al. 2024. An analysis of BPE vocabulary trimming in neural machine translation. In Proceedings of the Fifth Workshop on Insights from Negative Results in NLP, pages 48–50, Mexico City, Mexico. Association for Computational Linguistics.

Jiaxi Cui, Munan Ning, et al. 2024a. Chatlaw: A multi-agent collaborative legal assistant with knowledge graph enhanced mixture-of-experts large language model. Preprint, arXiv:2306.16092.

Yiming Cui, Ziqing Yang, and Xin Yao. 2024b. Efficient and effective text encoding for chinese llama and alpaca. Preprint, arXiv:2304.08177.

Tim Dettmers, Artidoro Pagnoni, et al. 2024. Qlora: efficient finetuning of quantized lms. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS ’23, Red Hook, NY, USA. Curran Associates Inc.

Abhimanyu Dubey, Abhinav Jauhari, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.

Alexander R. Fabbri, Wojciech Kryściński, et al. 2021. SummEval: Re-evaluating summarization evaluation. Transactions of the Association for Computational Linguistics, 9:391–409.

Pengju Gao, Tomohiro Yamasaki, and Kazunori Imoto. 2024. Ve-kd: Vocabulary-expansion knowledge-distillation for training smaller domain-specific language models. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 15046–15059.

Suchin Gururangan, Ana Marasović, et al. 2020. Don’t stop pretraining: Adapt language models to domains and tasks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8342–8360, Online. Association for Computational Linguistics.

Valentin Hofmann, Hinrich Schuetze, and Janet Pierre-humbert. 2022. An embarrassingly simple method to mitigate undesirable properties of pretrained language model tokenizers. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 385–393, Dublin, Ireland. Association for Computational Linguistics.

Jimin Hong, TaeHee Kim, et al. 2021. AVocaDo: Strategy for adapting vocabulary to downstream domain. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 4692–4700, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.

Edward J Hu, yelong shen, et al. 2022. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations.

Albert Q Jiang, Alexandre Sablayrolles, et al. 2023. Mistral 7b. arXiv preprint arXiv:2310.06825.

Qiao Jin, Bhuwan Dhingra, et al. 2019. Pubmedqa: A dataset for biomedical research question answering. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2567–2577.

Yanis Labrak, Adrien Bazoge, et al. 2024. BioMistral: A collection of open-source pretrained large language models for medical domains. In Findings of the Association for Computational Linguistics: ACL 2024, pages 5848–5864, Bangkok, Thailand. Association for Computational Linguistics.

Andrew Lampinen, Ishita Dasgupta, et al. 2022. Can language models learn from explanations in context? In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 537–563, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.

Anastasios Lamproudis, Aron Henriksson, et al. 2022. Vocabulary modifications for domain-adaptive pre-training of clinical language models. In Proceedings of the 15th International Joint Conference on
Biomedical Engineering Systems and Technologies (BIOSTEC 2022) - HEALTHINF, pages 180–188. INSTICC.

Haoran Lian, Yizhe Xiong, et al. 2025. Scaffold-bpe: Enhancing byte pair encoding for large language models with simple and effective scaffold token removal. Proceedings of the AAAI Conference on Artificial Intelligence, 39(23):24539–24548.

Chengyuan Liu, Shihang Wang, et al. 2024. Gold panning in vocabulary: An adaptive method for vocabulary expansion of domain-specific LLMs. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 7442–7459, Miami, Florida, USA. Association for Computational Linguistics.

Siyang Liu, Naihao Deng, et al. 2023. Task-adaptive tokenization: Enhancing long-form text generation efficacy in mental health and beyond. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 15264–15281, Singapore. Association for Computational Linguistics.

Sourab Mangrulkar, Sylvain Gugger, et al. 2022. Peft: State-of-the-art parameter-efficient fine-tuning methods. https://github.com/huggingface/peft.

Diego Mollá and Maria Elena Santiago-Martínez. 2011. Development of a corpus for evidence based medicine summarisation. In Proceedings of the Australasian Language Technology Association Workshop 2011, pages 86–94. Australian Language Technology Association.

Arijit Nag, Animesh Mukherjee, et al. 2024. Cost-performance optimization for processing low-resource language tasks using commercial LLMs. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 15681–15701, Miami, Florida, USA. Association for Computational Linguistics.

Arijit Nag, Bidisha Samanta, et al. 2023. Entropy-guided vocabulary augmentation of multilingual language models for low-resource tasks. In Findings of the Association for Computational Linguistics: ACL 2023, pages 8619–8629, Toronto, Canada. Association for Computational Linguistics.

Vincent Nguyen, Sarvnaz Karimi, and Zhenchang Xing. 2019. Investigating the effect of lexical segmentation in transformer-based models on medical datasets. In Proceedings of the 17th Annual Workshop of the Australasian Language Technology Association, pages 165–171.

Phillip Rust, Jonas Pfeiffer, et al. 2021. How good is your tokenizer? on the monolingual performance of multilingual language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 3118–3135, Online. Association for Computational Linguistics.

Abigail See, Peter J. Liu, and Christopher D. Manning. 2017. Get to the point: Summarization with pointer-generator networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1073–1083, Vancouver, Canada. Association for Computational Linguistics.

Yuqi Si, Jingqi Wang, et al. 2019. Enhancing clinical concept extraction with contextual embeddings. Journal of the American Medical Informatics Association, 26(11):1297–1304.

Luca Soldaini and Nazli Goharian. 2016. Quickumls: a fast, unsupervised approach for medical concept extraction. In MedIR workshop, SIGIR.

Atula Tejaswi, Nilesh Gupta, and Eunsol Choi. 2024. Exploring design choices for building language-specific LLMs. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 10485–10500, Miami, Florida, USA. Association for Computational Linguistics.

Hugo Touvron, Louis Martin, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288.

George Tsatsaronis, Georgios Balikas, et al. 2015. An overview of the bioasq large-scale biomedical semantic indexing and question answering competition. BMC bioinformatics, 16:1–28.

Dave Van Veen, Cara Van Uden, et al. 2024. Adapted large language models can outperform medical experts in clinical text summarization. Nature medicine, 30(4):1134–1142.

Benfeng Xu, Chunxu Zhao, et al. 2023. Retrieval-augmented domain adaptation of language models. In Proceedings of the 8th Workshop on Representation Learning for NLP (RepLANLP 2023), pages 54–64.

Jingjing Xu, Hao Zhou, et al. 2021. Vocabulary learning via optimal transport for neural machine translation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 7361–7373, Online. Association for Computational Linguistics.

Atuki Yamaguchi, Aline Villavicencio, and Nikolaos Aletras. 2024. How can we effectively expand the vocabulary of llms with 0.01 gb of target language text? arXiv preprint arXiv:2406.11477.

An Yang, Baosong Yang, et al. 2024. Qwen2 technical report. Preprint, arXiv:2407.10671.

Hongyi Yuan, Zheng Yuan, et al. 2022. BioBART: Pre-training and evaluation of a biomedical generative language model. In Proceedings of the 21st Workshop on Biomedical Language Processing, pages 97–109, Dublin, Ireland. Association for Computational Linguistics.
Nan Zhang, Yuesen Zhang, et al. 2023. Famesumsum: Investigating and improving faithfulness of medical summarization. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 10915–10931.

Weijia Zhang, Jia-Hong Huang, et al. 2025. Beyond relevant documents: A knowledge-intensive approach for query-focused summarization using large language models. In International Conference on Pattern Recognition, pages 89–104. Springer.

A Experimental Setup

Vocabulary Sizes. We report the vocabulary sizes along with the resultant fragment score in Table 8.

<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="2">Llama-2</th>
    <th colspan="2">Mistral</th>
    <th colspan="2">Llama-3.1</th>
    <th colspan="2">Qwen2</th>
  </tr>
  <tr>
    <th>VI</th>
    <th>FragSr</th>
    <th>VI</th>
    <th>FragSr</th>
    <th>VI</th>
    <th>FragSr</th>
    <th>VI</th>
    <th>FragSr</th>
  </tr>
  <tr>
    <td></td>
    <td colspan="8">PubMedQA</td>
  </tr>
  <tr>
    <td>CPT-Only</td>
    <td>32000</td>
    <td>2.53</td>
    <td>32000</td>
    <td>2.58</td>
    <td>128256</td>
    <td>2.29</td>
    <td>151646</td>
    <td>2.29</td>
  </tr>
  <tr>
    <td>MEDVOC</td>
    <td>38245</td>
    <td>1.48</td>
    <td>37642</td>
    <td>1.55</td>
    <td>129078</td>
    <td>1.21</td>
    <td>152441</td>
    <td>1.21</td>
  </tr>
  <tr>
    <td>MEDVOC-LLM</td>
    <td>34572</td>
    <td>1.66</td>
    <td>34219</td>
    <td>1.73</td>
    <td>128808</td>
    <td>1.25</td>
    <td>152131</td>
    <td>1.25</td>
  </tr>
  <tr>
    <td>ScafFix</td>
    <td>32200</td>
    <td>2.14</td>
    <td>32266</td>
    <td>2.20</td>
    <td>128456</td>
    <td>1.31</td>
    <td>152000</td>
    <td>1.28</td>
  </tr>
  <tr>
    <td></td>
    <td colspan="8">EBM</td>
  </tr>
  <tr>
    <td>CPT-Only</td>
    <td>32000</td>
    <td>2.65</td>
    <td>32000</td>
    <td>2.75</td>
    <td>128256</td>
    <td>2.33</td>
    <td>151646</td>
    <td>2.33</td>
  </tr>
  <tr>
    <td>MEDVOC</td>
    <td>42836</td>
    <td>1.48</td>
    <td>43908</td>
    <td>1.61</td>
    <td>131321</td>
    <td>1.12</td>
    <td>154092</td>
    <td>1.13</td>
  </tr>
  <tr>
    <td>MEDVOC-LLM</td>
    <td>34572</td>
    <td>1.66</td>
    <td>36252</td>
    <td>1.80</td>
    <td>130253</td>
    <td>1.22</td>
    <td>153468</td>
    <td>1.23</td>
  </tr>
  <tr>
    <td>ScafFix</td>
    <td>32150</td>
    <td>1.83</td>
    <td>32060</td>
    <td>2.54</td>
    <td>128456</td>
    <td>1.37</td>
    <td>152062</td>
    <td>1.35</td>
  </tr>
  <tr>
    <td></td>
    <td colspan="8">BioASQ</td>
  </tr>
  <tr>
    <td>CPT-Only</td>
    <td>32000</td>
    <td>2.98</td>
    <td>32000</td>
    <td>2.69</td>
    <td>128256</td>
    <td>2.44</td>
    <td>151646</td>
    <td>2.65</td>
  </tr>
  <tr>
    <td>MEDVOC</td>
    <td>43194</td>
    <td>1.93</td>
    <td>46502</td>
    <td>1.48</td>
    <td>133099</td>
    <td>1.24</td>
    <td>155518</td>
    <td>1.26</td>
  </tr>
  <tr>
    <td>MEDVOC-LLM</td>
    <td>57399</td>
    <td>2.18</td>
    <td>57206</td>
    <td>1.65</td>
    <td>130966</td>
    <td>1.30</td>
    <td>154612</td>
    <td>1.30</td>
  </tr>
  <tr>
    <td>ScafFix</td>
    <td>32300</td>
    <td>2.43</td>
    <td>32193</td>
    <td>2.39</td>
    <td>128506</td>
    <td>1.56</td>
    <td>152098</td>
    <td>1.44</td>
  </tr>
</table>

Table 8: The final vocabulary sizes (VI) along with the resultant fragment score (FragSr) observed on Medical OOV words. MEDVOC, MEDVOC-LLM, and ScafFix are the best vocabulary adaptation scheme which has the least vocabulary sizes and decent fragment score.

Prompts used. We use the prompt template inspired from ClinSumm (Van Veen et al., 2024) and Zhang et al. (2025) as shown in Table 9.

<table>
  <tr>
    <th>Prompt</th>
    <th>Query</th>
    <th>Document</th>
    <th>Summary</th>
  </tr>
  <tr>
    <td>You are a medical expert. You are given a query and query-relevant information as inputs. Your task is to summarize this information. The summary should be concise, include only non-redundant, query-relevant evidence, and be approximately 100 words long. Use the provided examples to guide word choice.</td>
    <td>--n lct examples concatenated using ##--</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Query [1]: {Test-Query}</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Document [1]: {Train-Source Document}</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Summary [1]: {Train-Summary}</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>##</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>–Test Example–</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Query: {Test-Query}</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Document: {Test-Source Document}</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Summary:</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
</table>

Table 9: Prompt Template used to prompt the language models for the task of query focused summarization.

<table>
  <tr>
    <th>Metric</th>
    <th>Llama-2</th>
    <th colspan="2">Subset-1</th>
    <th colspan="2">Subset-2</th>
  </tr>
  <tr>
    <td>Difficult-RS Concentration</td>
    <td></td>
    <td>9.78%</td>
    <td></td>
    <td>10.77%</td>
    <td></td>
  </tr>
  <tr>
    <td>Novel-RS Concentration</td>
    <td></td>
    <td>40.65%</td>
    <td></td>
    <td>41.92%</td>
    <td></td>
  </tr>
  <tr>
    <td>Rouge-LCS overlap between source and reference</td>
    <td></td>
    <td>38.17</td>
    <td></td>
    <td>35.65</td>
    <td></td>
  </tr>
  <tr>
    <th colspan="6">Llama-3.1</th>
  </tr>
  <tr>
    <td>Difficult-RS Concentration</td>
    <td></td>
    <td>7.47%</td>
    <td></td>
    <td>8.81%</td>
    <td></td>
  </tr>
  <tr>
    <td>Novel-RS Concentration</td>
    <td></td>
    <td>39.46%</td>
    <td></td>
    <td>42.78%</td>
    <td></td>
  </tr>
  <tr>
    <td>Rouge-LCS overlap between source and reference</td>
    <td></td>
    <td>39.52</td>
    <td></td>
    <td>34.27</td>
    <td></td>
  </tr>
</table>

Table 10: Difference of characteristics between instances where BASE has better Rouge-LCS than ScafFix (Subset-1) and instances where ScafFix has higher Rouge-LCS than BASE (Subset-2). The instances where BASE has better Rouge-LCS than ScafFix (Subset-1) have lesser values of Difficult-RS Concentration as well as Novel-RS Concentration, but higher values of Rouge-LCS overlap compared to Subset-2.

and CPT-Only in 7 out of 12 comparisons considered in the main text (\( Difficult_{SD} \), \( Difficult_{RS} \), \( Novel_{RS} \)). Here, we observe a similar trend with vocabulary adaptation not helping in case of low OOV and novelty concentration (specifically in BioASQ-\( \mathcal{S} \)) as we observed in main text (RQ6). The best performing continual pretraining strategy was End-to-End.

In case of Mistral, we observe that none of the vocabulary adaptation strategies improved over BASE and CPT-Only in one out of 12 comparisons considered in the main text (\( Difficult_{SD} \), \( Difficult_{RS} \), \( Novel_{RS} \)). Here, we observe a similar trend with vocabulary adaptation not helping in case of low OOV and novelty concentration (specifically in BioASQ-\( \mathcal{S} \)) as we observed in main text (RQ6). The best performing continual pretraining strategy was End-to-End for all the datasets except EBM.

B Results

Comparing Continual Pretraining Strategies for Llama-2 and Llama-3.1. In Table 11, we report the performance we observe using two different continual pretraining strategies on Llama-2 and in Table 12 we show performance for Llama-3.1 model. In case of Llama-2 the best pretraining strategy was Two-Stage and and for Llama-3.1 was End-To-End.

Mistral and Qwen results. We present the Mistral results in Table 13 and Qwen results in Table 14.

In case of Qwen2, we observe that at least one vocabulary adaptation strategies improve over BASE
<table>
  <tr>
    <th rowspan="2"> </th>
    <th colspan="4">End-To-End</th>
    <th colspan="4">Two-Stage</th>
  </tr>
  <tr>
    <th>TestFull</th>
    <th>DifficultRS</th>
    <th>NovelRS</th>
    <th>AllRS</th>
    <th>PubMedQX</th>
    <th>TestFull</th>
    <th>DifficultRS</th>
    <th>NovelRS</th>
    <th>AllRS</th>
  </tr>
  <tr>
    <td>BASE</td>
    <td>26.33</td>
    <td>23.53</td>
    <td>24.40</td>
    <td>19.78</td>
    <td>25.46</td>
    <td>27.45</td>
    <td>26.33</td>
    <td>23.53</td>
    <td>24.40</td>
    <td>19.78</td>
    <td>26.64</td>
    <td>27.45</td>
  </tr>
  <tr>
    <td>CPT-only</td>
    <td>27.12</td>
    <td>25.97</td>
    <td>28.00</td>
    <td>22.21</td>
    <td>27.86</td>
    <td>27.59</td>
    <td>27.11</td>
    <td>25.00</td>
    <td>26.92</td>
    <td>21.24</td>
    <td>24.07</td>
    <td>26.67</td>
  </tr>
  <tr>
    <td>MEDVOC</td>
    <td>26.50</td>
    <td>24.35</td>
    <td>22.88</td>
    <td>20.14</td>
    <td>24.59</td>
    <td>27.06</td>
    <td>26.39</td>
    <td>24.00</td>
    <td>24.82</td>
    <td>20.88</td>
    <td>22.22</td>
    <td>26.23</td>
  </tr>
  <tr>
    <td>MEDVOC-LLM</td>
    <td>26.90</td>
    <td>24.49</td>
    <td>26.67</td>
    <td>20.00</td>
    <td>25.27</td>
    <td>27.12</td>
    <td><b>27.30</b></td>
    <td><b>25.54</b></td>
    <td><b>27.45</b></td>
    <td>20.84</td>
    <td>21.43</td>
    <td>28.89</td>
  </tr>
  <tr>
    <td>ScafFix</td>
    <td><b>27.61</b></td>
    <td><b>26.32</b></td>
    <td><b>28.57</b></td>
    <td><b>22.22</b></td>
    <td><b>25.83</b></td>
    <td><b>28.57</b></td>
    <td>27.05</td>
    <td>25.00</td>
    <td>24.75</td>
    <td><b>21.30</b></td>
    <td><b>21.28</b></td>
    <td><b>30.19</b></td>
  </tr>
  <tr>
    <th colspan="13">EBM</th>
  </tr>
  <tr>
    <td>BASE</td>
    <td>18.56</td>
    <td>17.33</td>
    <td><b>16.00</b></td>
    <td>11.20</td>
    <td>14.95</td>
    <td><b>15.69</b></td>
    <td>18.56</td>
    <td>17.33</td>
    <td><b>16.00</b></td>
    <td>11.20</td>
    <td>14.95</td>
    <td><b>15.69</b></td>
  </tr>
  <tr>
    <td>CPT-only</td>
    <td><b>18.92</b></td>
    <td><b>18.16</b></td>
    <td>15.19</td>
    <td>11.77</td>
    <td>14.64</td>
    <td>14.00</td>
    <td>19.13</td>
    <td><b>19.05</b></td>
    <td>15.87</td>
    <td>11.77</td>
    <td>14.29</td>
    <td>13.95</td>
  </tr>
  <tr>
    <td>MEDVOC</td>
    <td>18.09</td>
    <td>18.18</td>
    <td>14.63</td>
    <td>11.84</td>
    <td>14.14</td>
    <td>13.74</td>
    <td>18.60</td>
    <td>18.05</td>
    <td>15.38</td>
    <td>12.03</td>
    <td>14.29</td>
    <td>14.29</td>
  </tr>
  <tr>
    <td>MEDVOC-LLM</td>
    <td>18.77</td>
    <td>17.20</td>
    <td>14.55</td>
    <td>12.50</td>
    <td>14.45</td>
    <td>15.27</td>
    <td><b>19.27</b></td>
    <td>17.54</td>
    <td>14.71</td>
    <td><b>12.90</b></td>
    <td><b>17.55</b></td>
    <td>14.81</td>
  </tr>
  <tr>
    <td>ScafFix</td>
    <td>18.67</td>
    <td>16.67</td>
    <td>15.39</td>
    <td>13.80</td>
    <td>15.27</td>
    <td>15.63</td>
    <td>18.65</td>
    <td>17.40</td>
    <td>14.71</td>
    <td>11.77</td>
    <td>15.47</td>
    <td>14.71</td>
  </tr>
  <tr>
    <th colspan="13">BioSNQ-M</th>
  </tr>
  <tr>
    <td>BASE</td>
    <td><b>28.50</b></td>
    <td>28.00</td>
    <td>27.27</td>
    <td><b>21.53</b></td>
    <td><b>26.71</b></td>
    <td>27.27</td>
    <td><b>28.50</b></td>
    <td>28.00</td>
    <td>27.27</td>
    <td><b>21.53</b></td>
    <td><b>26.71</b></td>
    <td>27.27</td>
  </tr>
  <tr>
    <td>CPT-only</td>
    <td>27.22</td>
    <td>28.57</td>
    <td>27.91</td>
    <td>20.31</td>
    <td>24.44</td>
    <td>27.91</td>
    <td>26.73</td>
    <td><b>29.41</b></td>
    <td>27.91</td>
    <td>19.80</td>
    <td>25.20</td>
    <td><b>28.57</b></td>
  </tr>
  <tr>
    <td>MEDVOC</td>
    <td>24.19</td>
    <td>26.32</td>
    <td>23.26</td>
    <td>16.53</td>
    <td>23.08</td>
    <td>26.09</td>
    <td>24.47</td>
    <td>28.07</td>
    <td>21.74</td>
    <td>16.06</td>
    <td>24.19</td>
    <td>24.57</td>
  </tr>
  <tr>
    <td>MEDVOC-LLM</td>
    <td>24.50</td>
    <td>28.00</td>
    <td>25.00</td>
    <td>18.90</td>
    <td>23.76</td>
    <td>26.38</td>
    <td>25.15</td>
    <td>27.59</td>
    <td>25.00</td>
    <td>16.85</td>
    <td>23.17</td>
    <td>24.69</td>
  </tr>
  <tr>
    <td>ScafFix</td>
    <td>26.16</td>
    <td><b>30.00</b></td>
    <td>29.79</td>
    <td>21.17</td>
    <td>21.84</td>
    <td><b>28.07</b></td>
    <td>26.00</td>
    <td>28.57</td>
    <td>28.57</td>
    <td>19.53</td>
    <td>24.10</td>
    <td>26.49</td>
  </tr>
  <tr>
    <th colspan="13">BioSNQ</th>
  </tr>
  <tr>
    <td>BASE</td>
    <td>32.12</td>
    <td>35.72</td>
    <td>26.32</td>
    <td><b>20.59</b></td>
    <td><b>33.33</b></td>
    <td><b>27.92</b></td>
    <td>32.12</td>
    <td>35.72</td>
    <td>26.12</td>
    <td>20.53</td>
    <td><b>33.33</b></td>
    <td><b>27.92</b></td>
  </tr>
  <tr>
    <td>CPT-only</td>
    <td>33.09</td>
    <td>29.35</td>
    <td>25.53</td>
    <td>19.64</td>
    <td>26.65</td>
    <td>24.12</td>
    <td><b>33.30</b></td>
    <td>30.33</td>
    <td>29.22</td>
    <td>20.00</td>
    <td>30.30</td>
    <td>21.92</td>
  </tr>
  <tr>
    <td>MEDVOC</td>
    <td>32.34</td>
    <td>30.26</td>
    <td>29.41</td>
    <td>20.29</td>
    <td>29.41</td>
    <td>22.47</td>
    <td>32.26</td>
    <td>27.40</td>
    <td>28.17</td>
    <td>18.18</td>
    <td>26.09</td>
    <td>23.53</td>
  </tr>
  <tr>
    <td>MEDVOC-LLM</td>
    <td>32.75</td>
    <td>34.52</td>
    <td>30.30</td>
    <td>29.29</td>
    <td>33.33</td>
    <td>24.47</td>
    <td>32.40</td>
    <td>30.33</td>
    <td><b>29.41</b></td>
    <td>20.29</td>
    <td>27.03</td>
    <td>21.53</td>
  </tr>
  <tr>
    <td>ScafFix</td>
    <td><b>33.34</b></td>
    <td><b>36.44</b></td>
    <td><b>31.25</b></td>
    <td>19.05</td>
    <td>27.03</td>
    <td>24.00</td>
    <td>32.88</td>
    <td><b>36.44</b></td>
    <td>29.29</td>
    <td><b>22.22</b></td>
    <td><b>33.33</b></td>
    <td>24.00</td>
  </tr>
</table>

Table 11: Performance comparison in terms of Rouge-L (R-L) between the two continual pretraining strategies of ‘End-to-End’ and ‘Two-Stage’ on Llama-2 7B model.

erence Summary (i.e., Difficult-RS Concentration), fraction of novel unigrams in reference summary (i.e., Novel-RS Concentration) and content overlap (measured by the standard metric Rouge-LCS) between source document and reference summary (i.e., Rouge-LCS overlap between source and reference). The differences between the two subsets in terms of these characteristics are compared in Table 10.

In terms of understanding the error, we made a key observation. The instances where BASE has better Rouge-LCS than ScafFix (Subset-1) have lesser values of Difficult-RS Concentration as well as Novel-RS Concentration, but higher values of Rouge-LCS overlap compared to Subset-2. Thus, for the instances that are less novel, have less OOV concentration, and are easier to infer, ScafFix is less helpful. This result from error analysis is also in line with the comparison of fine-grained settings (Table 6) reported in the main text, where we see general improvements for both Llama-2 and Llama-3.1 models for instances with higher OOV and higher novelty settings.

C Human Evaluation

We conducted our survey on Prolific platform4 where we hired 9 medical experts from the platform across globe. All the annotators were shown 10 random samples from a pool of 30 summaries where the order of summaries was randomized and blinded (annotators have no idea which summary came from which model). The median time to complete was set at 30 mins and the annotators were paid at the rate of 8 UK pounds per hour based on the amount of time they took. We did not collect any PII from the participants explicitly other than what was provided by the platform. The task was conducted using Google Forms, with participants being shown a consent notice beforehand. The results are shown in Figure 3.

Participation Criteria. The filtering criteria for participants were kept same as that of MEDVOC (Balde et al., 2024b):

1. Age: \( \geq 25 \),
2. Primary Language: English,
3. Highest education level completed: Graduate degree (MA/MSc/MPhil/other), Doctorate degree (PhD/other), and
4. Subject: Medicine, Health and Medicine, Biomedical Sciences.

Annotation Guidelines. The annotations were assessed across three key dimensions as outlined by Fabbri et al. (2021): Coherence, Relevance, and Factual Consistency.

Coherence evaluates the structural integrity of the summaries, focusing on whether the sentences are logically connected and contextually aligned. Relevance measures the informativeness of the
<table>
  <tr>
    <th rowspan="2"> </th>
    <th colspan="6">End-To-End</th>
    <th colspan="6">Two-Stage</th>
  </tr>
  <tr>
    <th>TestFull</th>
    <th>DifficultESO</th>
    <th>DifficultRGS</th>
    <th>NovelESO</th>
    <th>PubMedQA</th>
    <th>AllESO</th>
    <th>AllRGS</th>
    <th>TestFull</th>
    <th>DifficultESO</th>
    <th>DifficultRGS</th>
    <th>NovelESO</th>
    <th>AllESO</th>
    <th>AllRGS</th>
  </tr>
  <tr>
    <td>BASE</td>
    <td><b>28.10</b></td>
    <td>24.46</td>
    <td>26.87</td>
    <td>18.87</td>
    <td><b>28.89</b></td>
    <td>29.21</td>
    <td><b>28.10</b></td>
    <td>24.46</td>
    <td>26.87</td>
    <td>18.87</td>
    <td>29.23</td>
    <td>32.10</td>
  </tr>
  <tr>
    <td>CPT-only</td>
    <td>26.62</td>
    <td>24.52</td>
    <td><b>27.08</b></td>
    <td>21.07</td>
    <td>25.93</td>
    <td>26.74</td>
    <td>27.05</td>
    <td><b>27.71</b></td>
    <td>26.97</td>
    <td>19.64</td>
    <td><b>26.71</b></td>
    <td>28.65</td>
  </tr>
  <tr>
    <td>MEDVOC</td>
    <td>27.86</td>
    <td>24.68</td>
    <td>26.23</td>
    <td>21.87</td>
    <td>25.97</td>
    <td><b>31.12</b></td>
    <td>27.17</td>
    <td>25.76</td>
    <td><b>27.12</b></td>
    <td>19.25</td>
    <td>28.06</td>
    <td>29.62</td>
  </tr>
  <tr>
    <td>MEDVOC-LLM</td>
    <td>27.69</td>
    <td><b>26.51</b></td>
    <td>26.67</td>
    <td>21.74</td>
    <td>25.53</td>
    <td>30.43</td>
    <td>27.15</td>
    <td>25.19</td>
    <td>26.51</td>
    <td>19.86</td>
    <td>28.45</td>
    <td>28.51</td>
  </tr>
  <tr>
    <td>ScafFix</td>
    <td>27.67</td>
    <td>23.85</td>
    <td>25.00</td>
    <td><b>22.22</b></td>
    <td>25.00</td>
    <td>29.16</td>
    <td>27.25</td>
    <td>26.37</td>
    <td>26.42</td>
    <td><b>21.16</b></td>
    <td>28.16</td>
    <td><b>28.41</b></td>
  </tr>
  <tr>
    <td colspan="14">EBM</td>
  </tr>
  <tr>
    <td>BASE</td>
    <td>20.04</td>
    <td>15.79</td>
    <td>14.54</td>
    <td>11.37</td>
    <td>15.62</td>
    <td>15.18</td>
    <td>20.04</td>
    <td>15.79</td>
    <td>14.54</td>
    <td>11.37</td>
    <td>15.62</td>
    <td>15.18</td>
  </tr>
  <tr>
    <td>CPT-only</td>
    <td>20.13</td>
    <td>16.99</td>
    <td>17.15</td>
    <td>13.33</td>
    <td>13.79</td>
    <td>16.67</td>
    <td>20.20</td>
    <td>17.24</td>
    <td>14.94</td>
    <td>13.90</td>
    <td>14.81</td>
    <td>13.82</td>
  </tr>
  <tr>
    <td>MEDVOC</td>
    <td>20.11</td>
    <td>17.93</td>
    <td>16.03</td>
    <td>12.17</td>
    <td>15.86</td>
    <td>15.87</td>
    <td>20.35</td>
    <td><b>17.65</b></td>
    <td>15.56</td>
    <td>13.34</td>
    <td><b>15.26</b></td>
    <td><b>15.78</b></td>
  </tr>
  <tr>
    <td>MEDVOC-LLM</td>
    <td>20.75</td>
    <td><b>18.39</b></td>
    <td>16.40</td>
    <td>13.04</td>
    <td><b>16.13</b></td>
    <td><b>17.05</b></td>
    <td>20.42</td>
    <td><b>17.65</b></td>
    <td>15.63</td>
    <td><b>14.82</b></td>
    <td>15.79</td>
    <td>15.38</td>
  </tr>
  <tr>
    <td>ScafFix</td>
    <td><b>20.79</b></td>
    <td>16.18</td>
    <td><b>17.17</b></td>
    <td>13.80</td>
    <td>13.59</td>
    <td>14.29</td>
    <td><b>20.50</b></td>
    <td>17.07</td>
    <td><b>16.85</b></td>
    <td>13.04</td>
    <td>16.13</td>
    <td>14.76</td>
  </tr>
  <tr>
    <td colspan="14">BioASQ-M</td>
  </tr>
  <tr>
    <td>BASE</td>
    <td><b>29.28</b></td>
    <td>28.57</td>
    <td>26.67</td>
    <td><b>21.51</b></td>
    <td>25.54</td>
    <td>23.77</td>
    <td><b>29.28</b></td>
    <td>28.57</td>
    <td>26.67</td>
    <td><b>21.51</b></td>
    <td><b>25.53</b></td>
    <td>23.77</td>
  </tr>
  <tr>
    <td>CPT-only</td>
    <td>27.56</td>
    <td>28.57</td>
    <td>26.09</td>
    <td>19.84</td>
    <td>25.54</td>
    <td>23.53</td>
    <td>27.25</td>
    <td>28.57</td>
    <td><b>29.79</b></td>
    <td>19.14</td>
    <td>24.56</td>
    <td><b>27.73</b></td>
  </tr>
  <tr>
    <td>MEDVOC</td>
    <td>27.71</td>
    <td>28.57</td>
    <td>27.27</td>
    <td>20.46</td>
    <td>24.39</td>
    <td>25.86</td>
    <td>27.18</td>
    <td>30.00</td>
    <td>26.93</td>
    <td>19.33</td>
    <td>24.56</td>
    <td>26.84</td>
  </tr>
  <tr>
    <td>MEDVOC-LLM</td>
    <td>27.45</td>
    <td>29.17</td>
    <td>27.27</td>
    <td>19.05</td>
    <td>25.00</td>
    <td>26.09</td>
    <td>27.92</td>
    <td><b>30.43</b></td>
    <td>29.03</td>
    <td>20.97</td>
    <td>25.00</td>
    <td>27.36</td>
  </tr>
  <tr>
    <td>ScafFix</td>
    <td>28.91</td>
    <td><b>30.00</b></td>
    <td><b>27.59</b></td>
    <td>21.23</td>
    <td><b>25.89</b></td>
    <td><b>26.45</b></td>
    <td>27.86</td>
    <td>28.13</td>
    <td>28.57</td>
    <td>20.94</td>
    <td>24.10</td>
    <td>27.38</td>
  </tr>
  <tr>
    <td colspan="14">BioASQ-S</td>
  </tr>
  <tr>
    <td>BASE</td>
    <td>35.25</td>
    <td>27.03</td>
    <td>29.60</td>
    <td>18.18</td>
    <td>30.15</td>
    <td><b>27.92</b></td>
    <td>35.25</td>
    <td>27.03</td>
    <td>29.60</td>
    <td>18.18</td>
    <td>30.15</td>
    <td>27.92</td>
  </tr>
  <tr>
    <td>CPT-only</td>
    <td>37.60</td>
    <td><b>42.42</b></td>
    <td>30.04</td>
    <td>18.67</td>
    <td><b>42.02</b></td>
    <td>31.58</td>
    <td>36.01</td>
    <td>29.41</td>
    <td>24.70</td>
    <td>17.14</td>
    <td>31.79</td>
    <td>28.57</td>
  </tr>
  <tr>
    <td>MEDVOC</td>
    <td>37.00</td>
    <td>31.25</td>
    <td>27.03</td>
    <td>20.29</td>
    <td>30.38</td>
    <td>28.92</td>
    <td>37.01</td>
    <td>32.43</td>
    <td>29.79</td>
    <td>18.18</td>
    <td>32.43</td>
    <td>31.11</td>
  </tr>
  <tr>
    <td>MEDVOC-LLM</td>
    <td>27.43</td>
    <td>35.09</td>
    <td>29.34</td>
    <td>20.29</td>
    <td>30.00</td>
    <td>30.37</td>
    <td><b>37.15</b></td>
    <td><b>37.84</b></td>
    <td><b>32.89</b></td>
    <td>20.29</td>
    <td>31.01</td>
    <td>35.04</td>
  </tr>
  <tr>
    <td>ScafFix</td>
    <td><b>37.22</b></td>
    <td>37.04</td>
    <td><b>33.34</b></td>
    <td><b>20.34</b></td>
    <td>35.91</td>
    <td><b>35.02</b></td>
    <td>36.70</td>
    <td>34.15</td>
    <td>32.26</td>
    <td><b>20.90</b></td>
    <td><b>33.74</b></td>
    <td><b>33.35</b></td>
  </tr>
</table>

Table 12: Performance comparison in terms of Rouge-L (R-L) between the two continual pretraining strategies of ‘End-to-End’ and ‘Two-Stage’ on Llama-3.1 8B model.

summaries, considering the provided query and the context source document to judge the relevance. Faithfulness examines the accuracy of the stated facts, figures, and numerical data within the summaries, ensuring they can be directly verified against the input source. Notably, even if a summary presents accurate information, it is considered factually inconsistent if the claims cannot be substantiated solely by the given input. Relevance and Coherence are judged on a scale of 1-5; whereas factual consistency is binary (Yes/No).

To make the guidelines clear, we provide few examples of positive and negative examples to make people aware of what is a relevant, coherent, and factually consistent document vs what is not. The pdf is present in the github codebase.

D Representative Examples

We present examples from PubMedQA and EBM datasets where vocabulary adaptation strategy Scaf-Fix helped improve the performance over BASE in higher OOV concentration settings from Llama-2 models.
<table>
  <tr>
    <th rowspan="2"> </th>
    <th colspan="6">End-To-End</th>
    <th colspan="6">Two-Stage</th>
  </tr>
  <tr>
    <th>TestFull</th>
    <th>DifficultLSD</th>
    <th>DifficultLSS</th>
    <th>NovelLSS</th>
    <th>AllLSS</th>
    <th>AllLRS</th>
    <th>TestFull</th>
    <th>DifficultLSD</th>
    <th>DifficultLSS</th>
    <th>NovelLSS</th>
    <th>AllLSD</th>
    <th>AllLRS</th>
  </tr>
  <tr>
    <td>BASE</td>
    <td>25.40</td>
    <td>27.04</td>
    <td>28.32</td>
    <td>19.48</td>
    <td>22.97</td>
    <td>29.29</td>
    <td>25.40</td>
    <td>27.04</td>
    <td>28.32</td>
    <td>19.48</td>
    <td>22.97</td>
    <td>29.29</td>
  </tr>
  <tr>
    <td>CPT-Only</td>
    <td>25.00</td>
    <td>25.00</td>
    <td>25.23</td>
    <td>19.14</td>
    <td>24.64</td>
    <td>27.80</td>
    <td>25.51</td>
    <td>25.28</td>
    <td>26.67</td>
    <td>19.84</td>
    <td>24.81</td>
    <td>28.37</td>
  </tr>
  <tr>
    <td>MEDVOC</td>
    <td>24.49</td>
    <td>24.09</td>
    <td>26.67</td>
    <td>17.56</td>
    <td>23.88</td>
    <td>28.21</td>
    <td>24.64</td>
    <td>24.99</td>
    <td>25.01</td>
    <td>18.85</td>
    <td>22.79</td>
    <td>26.49</td>
  </tr>
  <tr>
    <td>MEDVOC-LLM</td>
    <td><b>25.45</b></td>
    <td>25.05</td>
    <td>24.55</td>
    <td>19.04</td>
    <td>24.12</td>
    <td>26.62</td>
    <td>24.47</td>
    <td>24.06</td>
    <td>24.25</td>
    <td>19.20</td>
    <td>23.49</td>
    <td>28.10</td>
  </tr>
  <tr>
    <td>ScafFix</td>
    <td>25.40</td>
    <td>24.04</td>
    <td>25.27</td>
    <td>18.67</td>
    <td>24.62</td>
    <td>27.92</td>
    <td>24.56</td>
    <td>24.78</td>
    <td>24.85</td>
    <td>17.79</td>
    <td>23.14</td>
    <td>28.29</td>
  </tr>
  <tr>
    <td>BASE</td>
    <td><b>17.43</b></td>
    <td><b>17.86</b></td>
    <td><b>17.06</b></td>
    <td>11.49</td>
    <td><b>16.87</b></td>
    <td>12.33</td>
    <td><b>17.43</b></td>
    <td><b>17.86</b></td>
    <td><b>17.06</b></td>
    <td>11.49</td>
    <td><b>16.87</b></td>
    <td>13.33</td>
  </tr>
  <tr>
    <td>CPT-Only</td>
    <td>17.15</td>
    <td>17.70</td>
    <td>16.49</td>
    <td>12.12</td>
    <td>15.85</td>
    <td>13.79</td>
    <td>17.39</td>
    <td>17.82</td>
    <td>15.59</td>
    <td>11.32</td>
    <td>14.29</td>
    <td>12.77</td>
  </tr>
  <tr>
    <td>MEDVOC</td>
    <td>17.63</td>
    <td>16.09</td>
    <td>14.04</td>
    <td>11.24</td>
    <td>15.46</td>
    <td>12.99</td>
    <td>17.10</td>
    <td>15.83</td>
    <td>14.61</td>
    <td>13.01</td>
    <td>15.38</td>
    <td>13.70</td>
  </tr>
  <tr>
    <td>MEDVOC-LLM</td>
    <td>17.19</td>
    <td>17.40</td>
    <td>14.77</td>
    <td>11.11</td>
    <td>15.19</td>
    <td>13.33</td>
    <td>16.85</td>
    <td>16.22</td>
    <td>15.48</td>
    <td>12.66</td>
    <td>15.19</td>
    <td>12.50</td>
  </tr>
  <tr>
    <td>ScafFix</td>
    <td>16.80</td>
    <td>17.19</td>
    <td>14.98</td>
    <td>11.19</td>
    <td>15.22</td>
    <td>13.11</td>
    <td>17.21</td>
    <td>16.87</td>
    <td>13.30</td>
    <td>11.99</td>
    <td>13.33</td>
    <td>12.90</td>
  </tr>
  <tr>
    <td>BASE</td>
    <td><b>25.97</b></td>
    <td><b>28.21</b></td>
    <td><b>25.64</b></td>
    <td><b>21.88</b></td>
    <td><b>23.21</b></td>
    <td><b>26.67</b></td>
    <td><b>25.97</b></td>
    <td><b>28.21</b></td>
    <td><b>25.64</b></td>
    <td><b>21.88</b></td>
    <td><b>26.23</b></td>
    <td><b>26.67</b></td>
  </tr>
  <tr>
    <td>CPT-Only</td>
    <td>22.73</td>
    <td>25.64</td>
    <td>25.00</td>
    <td>21.05</td>
    <td>23.38</td>
    <td>25.33</td>
    <td>23.53</td>
    <td>27.03</td>
    <td>24.56</td>
    <td>21.54</td>
    <td>23.26</td>
    <td>24.56</td>
  </tr>
  <tr>
    <td>MEDVOC</td>
    <td>21.54</td>
    <td>23.68</td>
    <td>17.02</td>
    <td>16.67</td>
    <td>22.20</td>
    <td>21.43</td>
    <td>21.95</td>
    <td>25.30</td>
    <td>15.38</td>
    <td>17.02</td>
    <td>22.95</td>
    <td>22.22</td>
  </tr>
  <tr>
    <td>MEDVOC-LLM</td>
    <td>22.73</td>
    <td>24.69</td>
    <td>20.00</td>
    <td>16.18</td>
    <td>23.08</td>
    <td>22.73</td>
    <td>22.64</td>
    <td>26.67</td>
    <td>19.23</td>
    <td>17.91</td>
    <td>24.00</td>
    <td>22.86</td>
  </tr>
  <tr>
    <td>ScafFix</td>
    <td>22.73</td>
    <td>24.39</td>
    <td>19.35</td>
    <td>18.52</td>
    <td>22.58</td>
    <td>23.81</td>
    <td>20.69</td>
    <td>23.53</td>
    <td>17.02</td>
    <td>16.33</td>
    <td>22.22</td>
    <td>21.28</td>
  </tr>
  <tr>
    <td>BASE</td>
    <td><b>35.44</b></td>
    <td><b>53.59</b></td>
    <td><b>40.00</b></td>
    <td><b>28.57</b></td>
    <td><b>40.00</b></td>
    <td><b>35.44</b></td>
    <td><b>53.59</b></td>
    <td><b>40.00</b></td>
    <td><b>28.57</b></td>
    <td><b>40.00</b></td>
    <td><b>28.57</b></td>
    <td><b>40.00</b></td>
  </tr>
  <tr>
    <td>CPT-Only</td>
    <td>28.57</td>
    <td>31.01</td>
    <td>23.58</td>
    <td>22.22</td>
    <td>28.57</td>
    <td>25.81</td>
    <td>29.63</td>
    <td>34.14</td>
    <td>28.57</td>
    <td>21.15</td>
    <td>32.26</td>
    <td>28.57</td>
  </tr>
  <tr>
    <td>MEDVOC</td>
    <td>28.57</td>
    <td>31.16</td>
    <td>26.32</td>
    <td>19.15</td>
    <td>27.03</td>
    <td>27.78</td>
    <td>27.78</td>
    <td>29.61</td>
    <td>25.93</td>
    <td>19.61</td>
    <td>28.00</td>
    <td>23.81</td>
  </tr>
  <tr>
    <td>MEDVOC-LLM</td>
    <td>28.17</td>
    <td>35.19</td>
    <td>27.91</td>
    <td>20.22</td>
    <td>35.90</td>
    <td>29.63</td>
    <td>27.27</td>
    <td>34.22</td>
    <td>26.09</td>
    <td>20.51</td>
    <td>30.30</td>
    <td>26.67</td>
  </tr>
  <tr>
    <td>ScafFix</td>
    <td>30.00</td>
    <td>35.52</td>
    <td>29.63</td>
    <td>25.00</td>
    <td>32.00</td>
    <td>33.39</td>
    <td>24.69</td>
    <td>31.95</td>
    <td>24.24</td>
    <td>20.00</td>
    <td>22.22</td>
    <td>25.00</td>
  </tr>
</table>

Table 13: Performance comparison in terms of Rouge-L (R-L) between the two continual pretraining strategies of 'End-to-End' and 'Two-Stage' on Mistral 7B model.

<table>
  <tr>
    <th rowspan="2"> </th>
    <th colspan="6">End-To-End</th>
    <th colspan="6">Two-Stage</th>
  </tr>
  <tr>
    <th>TestFull</th>
    <th>DifficultLSD</th>
    <th>DifficultLSS</th>
    <th>NovelLSS</th>
    <th>AllLSS</th>
    <th>AllLRS</th>
    <th>TestFull</th>
    <th>DifficultLSD</th>
    <th>DifficultLSS</th>
    <th>NovelLSS</th>
    <th>AllLSD</th>
    <th>AllLRS</th>
  </tr>
  <tr>
    <td>BASE</td>
    <td>24.00</td>
    <td>22.22</td>
    <td>24.14</td>
    <td>18.82</td>
    <td>23.74</td>
    <td>29.00</td>
    <td>24.00</td>
    <td>22.22</td>
    <td>24.14</td>
    <td>18.82</td>
    <td>23.74</td>
    <td>30.00</td>
  </tr>
  <tr>
    <td>CPT-only</td>
    <td>24.65</td>
    <td><b>27.85</b></td>
    <td>25.29</td>
    <td>17.50</td>
    <td><b>28.85</b></td>
    <td>25.81</td>
    <td>24.73</td>
    <td>23.88</td>
    <td>24.78</td>
    <td>17.65</td>
    <td><b>25.41</b></td>
    <td>25.64</td>
  </tr>
  <tr>
    <td>MEDVOC</td>
    <td>24.59</td>
    <td>23.08</td>
    <td><b>25.93</b></td>
    <td>18.37</td>
    <td>23.21</td>
    <td>26.32</td>
    <td>24.89</td>
    <td>25.45</td>
    <td>26.20</td>
    <td>18.18</td>
    <td>23.57</td>
    <td>25.00</td>
  </tr>
  <tr>
    <td>MEDVOC-LLM</td>
    <td><b>25.00</b></td>
    <td>25.00</td>
    <td>25.90</td>
    <td>19.50</td>
    <td>28.83</td>
    <td>26.67</td>
    <td><b>25.00</b></td>
    <td><b>27.06</b></td>
    <td><b>16.87</b></td>
    <td>23.47</td>
    <td>27.60</td>
    <td>25.00</td>
  </tr>
  <tr>
    <td>ScafFix</td>
    <td>24.66</td>
    <td>25.58</td>
    <td>25.65</td>
    <td>18.60</td>
    <td>23.81</td>
    <td>25.44</td>
    <td>25.00</td>
    <td>26.32</td>
    <td><b>18.87</b></td>
    <td>25.00</td>
    <td>26.32</td>
    <td>25.00</td>
  </tr>
  <tr>
    <td>BASE</td>
    <td><b>17.52</b></td>
    <td>16.49</td>
    <td>14.04</td>
    <td>11.43</td>
    <td>16.07</td>
    <td>14.46</td>
    <td><b>17.52</b></td>
    <td>16.49</td>
    <td>14.04</td>
    <td>11.43</td>
    <td>16.07</td>
    <td>14.46</td>
  </tr>
  <tr>
    <td>CPT-only</td>
    <td>17.60</td>
    <td>15.38</td>
    <td>15.17</td>
    <td>11.86</td>
    <td>14.62</td>
    <td>13.91</td>
    <td><b>17.80</b></td>
    <td>14.93</td>
    <td>14.89</td>
    <td>11.51</td>
    <td>14.77</td>
    <td>13.95</td>
  </tr>
  <tr>
    <td>MEDVOC</td>
    <td>17.54</td>
    <td>16.13</td>
    <td>16.13</td>
    <td>12.31</td>
    <td>16.67</td>
    <td>15.62</td>
    <td>17.43</td>
    <td>15.87</td>
    <td>15.47</td>
    <td>12.50</td>
    <td><b>16.75</b></td>
    <td>15.15</td>
  </tr>
  <tr>
    <td>MEDVOC-LLM</td>
    <td>17.34</td>
    <td>16.13</td>
    <td>14.70</td>
    <td>12.60</td>
    <td>16.16</td>
    <td>15.15</td>
    <td>17.65</td>
    <td><b>17.93</b></td>
    <td>15.39</td>
    <td>11.76</td>
    <td>16.39</td>
    <td>15.15</td>
  </tr>
  <tr>
    <td>ScafFix</td>
    <td><b>17.76</b></td>
    <td><b>17.24</b></td>
    <td><b>17.64</b></td>
    <td>13.55</td>
    <td>18.82</td>
    <td>17.68</td>
    <td>17.67</td>
    <td>17.24</td>
    <td><b>16.87</b></td>
    <td>12.04</td>
    <td>16.07</td>
    <td><b>17.14</b></td>
  </tr>
  <tr>
    <td>BASE</td>
    <td><b>25.45</b></td>
    <td>30.00</td>
    <td><b>27.27</b></td>
    <td>19.67</td>
    <td>26.09</td>
    <td><b>27.73</b></td>
    <td><b>25.45</b></td>
    <td>30.00</td>
    <td><b>27.27</b></td>
    <td>19.67</td>
    <td><b>26.09</b></td>
    <td><b>27.73</b></td>
  </tr>
  <tr>
    <td>CPT-only</td>
    <td>23.88</td>
    <td>28.57</td>
    <td>26.10</td>
    <td>18.52</td>
    <td>25.25</td>
    <td>26.90</td>
    <td>24.56</td>
    <td>28.57</td>
    <td>25.00</td>
    <td>17.86</td>
    <td>25.35</td>
    <td>26.90</td>
  </tr>
  <tr>
    <td>MEDVOC</td>
    <td>25.00</td>
    <td><b>30.20</b></td>
    <td>26.67</td>
    <td><b>20.00</b></td>
    <td><b>26.67</b></td>
    <td>26.85</td>
    <td>24.49</td>
    <td>28.57</td>
    <td>25.00</td>
    <td>19.05</td>
    <td>24.70</td>
    <td>26.32</td>
  </tr>
  <tr>
    <td>MEDVOC-LLM</td>
    <td>24.30</td>
    <td>28.57</td>
    <td>25.71</td>
    <td>19.23</td>
    <td>25.35</td>
    <td>27.27</td>
    <td>24.34</td>
    <td>28.57</td>
    <td>26.67</td>
    <td><b>20.00</b></td>
    <td>25.00</td>
    <td>26.80</td>
  </tr>
  <tr>
    <td>ScafFix</td>
    <td>24.49</td>
    <td>28.24</td>
    <td>26.23</td>
    <td>19.67</td>
    <td>25.00</td>
    <td>27.20</td>
    <td>24.69</td>
    <td>28.57</td>
    <td>26.67</td>
    <td><b>20.00</b></td>
    <td>25.00</td>
    <td>27.27</td>
  </tr>
  <tr>
    <td>BASE</td>
    <td><b>36.19</b></td>
    <td><b>45.31</b></td>
    <td><b>45.24</b></td>
    <td><b>28.57</b></td>
    <td><b>41.89</b></td>
    <td><b>43.17</b></td>
    <td><b>36.19</b></td>
    <td><b>45.31</b></td>
    <td><b>45.24</b></td>
    <td><b>28.57</b></td>
    <td><b>41.89</b></td>
    <td><b>43.17</b></td>
  </tr>
  <tr>
    <td>CPT-only</td>
    <td>32.43</td>
    <td>37.89</td>
    <td>28.53</td>
    <td>24.49</td>
    <td>32.18</td>
    <td>31.59</td>
    <td>31.59</td>
    <td>32.57</td>
    <td>25.00</td>
    <td>34.46</td>
    <td>29.90</td>
    <td>30.25</td>
  </tr>
  <tr>
    <td>MEDVOC</td>
    <td>31.91</td>
    <td>35.56</td>
    <td>32.46</td>
    <td>24.62</td>
    <td>32.95</td>
    <td>31.28</td>
    <td>31.59</td>
    <td>33.69</td>
    <td>31.86</td>
    <td>25.00</td>
    <td>33.81</td>
    <td>30.25</td>
  </tr>
  <tr>
    <td>MEDVOC-LLM</td>
    <td>32.14</td>
    <td>30.49</td>
    <td>30.94</td>
    <td>26.32</td>
    <td>33.81</td>
    <td>30.77</td>
    <td>32.14</td>
    <td>37.97</td>
    <td>31.84</td>
    <td>24.56</td>
    <td>36.18</td>
    <td>34.29</td>
  </tr>
  <tr>
    <td>ScafFix</td>
    <td>32.00</td>
    <td>39.36</td>
    <td>36.94</td>
    <td>26.67</td>
    <td>37.23</td>
    <td>31.01</td>
    <td>31.58</td>
    <td>38.28</td>
    <td>32.13</td>
    <td>26.00</td>
    <td>36.04</td>
    <td>29.10</td>
  </tr>
</table>

Table 14: Performance comparison in terms of Rouge-L (R-L) between the two continual pretraining strategies of 'End-to-End' and 'Two-Stage' on Qwen2 7B model.
<table>
  <tr>
    <th>Input</th>
  </tr>
  <tr>
    <td><b>Query:</b> What medication best prevents migraine in children?</td>
  </tr>
  <tr>
    <td><b>PubMed Abstract:</b> In a prospective study we compared propranolol, placebo, and self-hypnosis in the treatment of juvenile classic migraine. Children aged 6 to 12 years with classic migraine who had no previous specific treatment were randomized into propranolol (0.5 mg/kg/d) or placebo groups for a 3-month period. In these groups over for 3 months, a third group of 6-months, in which each child was taught self-hypnosis and used it for 3 months. Twenty-eight patients completed the entire study. The mean number of headaches per child for 3 months during the placebo period was 13.3 compared with 14.9 during the propranolol period and 5.8 during the self-hypnosis period. Statistical analysis showed a significant association between decrease in headache frequency and self-hypnosis training (P = .045). There was no significant change in subjective or objective measures of headache severity with either therapy.</td>
  </tr>
  <tr>
    <td><b>Reference Summary (OOV Concentration: 7.7%)</b></td>
  </tr>
  <tr>
    <td>A comparative randomized controlled trial with multiple crossovers involving 33 children found that a self-hypnosis placebo decreased mean headache frequency from 13.3 per 3-month interval to 5.8 (P=.045), but found propranolol no different than placebo. Propranolol was also studied in a 3-armed randomized controlled trial in comparison with flunarizine—a drug likely to be efficacious and placebo.</td>
  </tr>
  <tr>
    <td><b>BASE Summary (Rouge-L: 24.10)</b></td>
  </tr>
  <tr>
    <td>Self-hypnosis was associated with a significant decrease in headache frequency in a single randomized controlled trial of 28 children.</td>
  </tr>
  <tr>
    <td><b>ScaFix Summary (Rouge-L: 40.10)</b></td>
  </tr>
  <tr>
    <td>In a single randomized controlled trial with crossover design in 28 children, self-hypnosis reduced headache frequency from a mean of 13.3 attacks/month to 5.8 attacks/month vs. no change for placebo (P =.045).</td>
  </tr>
</table>

Table 15: Example from EBM dataset comparing summary generated using vocabulary adaptation method ScaFix with BASE.

<table>
  <tr>
    <th>Input</th>
  </tr>
  <tr>
    <td><b>Query:</b> Should serum pancreatic lipase replace serum amylase as a biomarker of acute pancreatitis?</td>
  </tr>
  <tr>
    <td><b>PubMed Abstract:</b> Serum pancreatic lipase may improve the diagnosis of pancreatitis compared to serum amylase. Both enzymes have been measured simultaneously at our hospital allowing for a comparison of their diagnostic accuracy. Seventeen thousand five hundred and thirty-one measurements of either serum amylase and or serum pancreatic lipase were made ... One thousand eight hundred and twenty-five patients had either elevated serum amylase and or serum pancreatic lipase. The medical records coded for pancreatitis in a further 55 whose enzymes were not elevated. Three hundred and twenty of these had radiological evidence of acute pancreatitis. Receiver operator characteristic analysis of the initial data from patients seen in the Emergency Department showed improved diagnostic accuracy for serum pancreatic lipase (area under the curve (AUC) 0.948) compared with serum amylase (AUC, 0.906, P<0.05) ...</td>
  </tr>
  <tr>
    <td><b>Reference Summary (OOV Concentration: 47.09%)</b></td>
  </tr>
  <tr>
    <td>It is concluded that serum pancreatic lipase is a more accurate biomarker of acute pancreatitis than serum amylase.</td>
  </tr>
  <tr>
    <td><b>BASE Summary (Rouge-L: 29.17)</b></td>
  </tr>
  <tr>
    <td>Serum pancreatic lipase may improve the diagnosis of pancreatitis compared to serum amylase. Both enzymes have been measured simultaneously at our hospital allowing for a comparison of their diagnostic accuracy.</td>
  </tr>
  <tr>
    <td><b>ScaFix Summary (Rouge-L: 76.47)</b></td>
  </tr>
  <tr>
    <td>Serum pancreatic lipase is a more sensitive and specific biomarker of acute pancreatitis than serum amylase.</td>
  </tr>
</table>

Table 16: Example from PubMedQA dataset comparing summary generated using vocabulary adaptation method ScaFix with BASE. Interestingly, BASE model suffered from LEAD-bias (See et al., 2017) (picking sentences from the front of the document as the summary) and missed the biomarker word completely.