BioMistral: A Collection of Open-Source Pretrained Large Language Models for Medical Domains (2024)

Abstract: Large Language Models (LLMs) have demonstrated remarkable versatility in recent years, offering potential applications across specialized domains such as healthcare and medicine. Despite the availability of various open-source LLMs tailored for health contexts, adapting general-purpose LLMs to the medical domain presents significant challenges. In this paper, we introduce BioMistral, an open-source LLM tailored for the biomedical domain, utilizing Mistral as its foundation model and further pre-trained on PubMed Central. We conduct a comprehensive evaluation of BioMistral on a benchmark comprising 10 established medical question-answering (QA) tasks in English. We also explore lightweight models obtained through quantization and model merging approaches. Our results demonstrate BioMistral’s superior performance compared to existing open-source medical models and its competitive edge against proprietary counterparts. Finally, to address the limited availability of data beyond English and to assess the multilingual generalization of medical LLMs, we automatically translated and evaluated this benchmark into 7 other languages. This marks the first large-scale multilingual evaluation of LLMs in the medical domain. Datasets, multilingual evaluation benchmarks, scripts, and all the models obtained during our experiments are freely released.

heading: 1 Introduction
text: In the rapidly evolving landscape of Natural Language Processing (NLP), generative Large Language Models (LLMs) like ChatGPT (OpenAI, 2023) and Vicuna (Zheng et al., 2023) have revolutionized human-computer interactions, demonstrating remarkable versatility and advanced capabilities across various tasks and domains. These models exhibit human-like comprehension and reasoning, enabling them to tackle basic textual understanding as well as complex problem-solving tasks. The emergence of open-source LLMs such as
BLOOM (Workshop et al., 2023) and LLaMA (Touvron et al., 2023a) underscores the transformative potential of these models, facilitating their innovative use in specialized domains including medicine (Dave et al., 2023).
However, integrating LLMs into healthcare and medicine presents unique challenges and opportunities (He et al., 2023; Zhou et al., 2024). While preliminary adoption has opened new avenues for innovation, concerns about data privacy risks associated with proprietary models like MedPaLM-2 (Singhal et al., 2023b) and GPT-4 (Nori et al., 2023a) have arisen. The community’s interest in specialized LLMs for healthcare has led to initiatives like PMC-LLaMA (Wu et al., 2023) and MedAlpaca (Han et al., 2023). However, the adoption of open-source medical models has been limited, primarily due to the lack of lightweight models allowing commercial use with performance comparable to larger or proprietary models. To address this gap, there is a need to develop specialized models based on open-source foundation ones like GPT-NeoX (Black et al., 2022), LLaMa 2 (Touvron et al., 2023b), or Mistral (Jiang et al., 2023), and to optimize them for use on consumer-grade devices while maintaining performance.
In this work, we present BioMistral 7B, a specialized LLM tailored for the biomedical domain, derived from Mistral 7B Instruct v0.1 (Jiang et al., 2023) and further pre-trained on PubMed Central. Our contributions include:
1. The construction of BioMistral 7B, the first open-source Mistral-based model for the biomedical domain, including the analysis of different evaluation strategies such as fewshot in-context learning and supervised finetuning. 2. An original study with the introduction of a benchmark of 10 medical question-answering (QA) tasks in English, automatically translated into 7 other languages, facilitating assessment against existing state-of-the-art
5848
open-source medical LLMs, and shedding light on its multilingual potential and robustness across diverse linguistic contexts. 3. A large in-depth quantitative analysis of the model’s truthfulness and calibration in multilingual contexts. 4. A rigorous evaluation of a suite of lightweight models derived through various efficient quantization approaches. 5. An exploration of novel model merging techniques between Mistral 7B Instruct and BioMistral 7B models, allowing leveraging the commonsense reasoning skills of specialized and general-purpose LLMs.
All datasets, the multilingual benchmark, preprocessing scripts, and models are accessible on HuggingFace1 and GitHub2 under Apache 2.0 license.

heading: 2 Related Work
text: Generative LLMs with a generalist purpose, such as GPT-4 (OpenAI et al., 2023) and Gemini (Team et al., 2023), have demonstrated exceptional performance across various NLP tasks, including QA, text summarization, and language translation in zero- and few-shot scenarios. While these models firstly remain proprietary, the open-source movement has played a crucial role in democratizing LLMs, offering powerful capabilities to the broader community with Llama 2 (Touvron et al., 2023b), Vicuna (Zheng et al., 2023), Falcon (Almazrouei et al., 2023) and Mistral (Jiang et al., 2023) models.
The adaptation of LLMs to specialized domains often involves encoding domain-specific knowledge to fully leverage the models’ capabilities. This adaptation process has led to the development of several open-source models from pre-trained general domain models. Encoder-based models, such as BERT (Devlin et al., 2019), and encoderdecoder, such as T5 (Raffel et al., 2020), have been adapted for the medical domain (Lee et al., 2019; Gu et al., 2021; Yuan et al., 2022; Lu et al., 2022) but faced challenges with QA tasks. More recently, multiple LLMs based on decoder-only architecture have been developed for the medical domain, such as BioGPT (Luo et al., 2022) based on GPT-2 (Radford et al., 2019), ClinicalGPT (Wang et al., 2023), based on BLOOM-7B (Workshop et al., 2023), PMC-LLaMA (Wu et al., 2023), MedAlpaca (Han et al., 2023) and Chat-Doctor (Li
1https://huggingface.co/BioMistral 2https://github.com/BioMistral/BioMistral
et al., 2023), based on LLaMA (Touvron et al., 2023a), and BioMedGPT-LM-7B (Zhang et al., 2024) and MediTron-7B (Chen et al., 2023), adapted from Llama-2 (Touvron et al., 2023b). In contrast, proprietary medical LLMs like GPT-4 MedPrompt (Nori et al., 2023b) and Med-PALM 2 (Singhal et al., 2023b) face usability issues similar to general-purpose models. Nonetheless, they can be used as benchmark "target goals" for the development of domain-specific open-source models.

heading: 3 BioMistral
text: In this section, we present the modules that facilitated the construction of BioMistral 7B. We first develop our training corpus (Section 3.1) used during further pre-training. We then present the model adaptation method (Section 3.2). Finally, we discuss the approaches for model merging (Section 3.3) and expose the employed quantization strategies (Section 3.4).

heading: 3.1 Pre-training Dataset
text: For LLM adaptation to the medical domain, we selected the PMC Open Access Subset3 for its comprehensive and freely accessible collection of medical research papers. This choice is guided by the success demonstrated by PMC-LLaMA (Wu et al., 2023), PubMedBERT (Gu et al., 2021), and SciFive (Phan et al., 2021), which have showcased significant enhancements in language modeling for medical applications. Our focus lies on the Commercial Use Allowed subset, encompassing documents licensed under various Creative Commons licenses (CC0, CC BY, CC BY-SA, and CC BY-ND). This subset ensures the reusability of our model’s outputs, even for commercial purposes.
In the preprocessing phase, we aim to optimize the dataset for training efficiency while considering hardware limitations. Our pre-training objective involves further pre-training Mistral on a subsample of this corpus, targeting 1.5 epochs within the 20-hour limit of Jean Zay HPC. This decision aligns with insights from the Zephyr model (Tunstall et al., 2023), which suggests that observing 1.5 times the corpus adequately enhances model performance, with marginal benefits beyond this threshold. We then meticulously selected 3 billion tokens from this pre-processed PubMed Central corpus, corresponding to roughly 1.47 million documents. The dataset comprises primarily English documents (98.75% of the corpus), with the remain-
3https://www.ncbi.nlm.nih.gov/pmc/tools/openftlist/
ing portion encompassing 9 languages, including Dutch, German, French, and others. Our strategy emphasizes a multilingual dataset approach by prioritizing non-English documents, supplemented with English texts, to ensure a diverse and representative training dataset to meet our 3 billion token target. The raw textual documents undergo pre-processing using the Mistral tokenizer, which includes tokenization and normalization processes.

heading: 3.2 Model Adaptation
text: Training details We leverage Mistral 7B Instruct v0.1 (Jiang et al., 2023) as the base model for adaptation due to its design tailored for incorporating instructions in prompts and its capacity for fine-tuning across diverse tasks using limited datasets. Pre-training settings for BioMistral 7B largely align with Mistral 7B Instruct v0.1. For optimization, we employ the AdamW (Loshchilov and Hutter, 2019) optimizer alongside a cosine learning rate scheduler. Our model architecture inherits the standard transformer architecture from Mistral, including features such as Grouped-Query Attention (Ainslie et al., 2023), Sliding Window Attention (Beltagy et al., 2020) and Rolling Buffer Cache. We maintain an input context length of 2,048 tokens across all models, including the quantized versions (see Section 3.4), in conjunction with FlashAttention-2 (Dao, 2023). For optimization, we set the learning rate to 2×10−5 with no warmup, a weight decay of 0.01, a gradient accumulation of 2, and a batch size of 16 on the Jean-Zay HPC with 32 NVIDIA A100 80GB GPUs. This configuration allows for a total batch size of 1,024. Due to the model and the AdamW optimizer’s inability to fit on a single GPU with BF16 precision, we employ the Fully Sharded Data Parallel distributed learning framework (Zhao et al., 2023).
Improving batching To enhance pre-training efficiency, we introduce a post-tokenization grouping method. This method aggregates variable-sized sequences marked by an end-of-sequence token (
) to fill the model’s 2,048-token sequences without padding. This reduces the sequence count by 87.88%, subsequently accelerating epoch times. Refer to Appendix C for pseudo-code detailing the grouping method.

heading: 3.3 Model Merging
text: Pre-trained models may lose effectiveness when applied beyond their specific domains (Labrak et al., 2023). Traditionally, separate models were used
for each application (Guo et al., 2018), increasing complexity and costs. Recent studies suggest merging pre-trained models to enhance performance and out-of-domain generalization (Cha et al., 2021; Arpit et al., 2022; Wortsman et al., 2022; Jin et al., 2023; Ilharco et al., 2023). Merging involves combining multiple model parameters without additional training. Methods include averaging model weights or considering permutation invariance (Ilharco et al., 2022; Choshen et al., 2022; Singh and Jaggi, 2020; Ainsworth et al., 2023).
Among these methods, we can cite TIES (Yadav et al., 2023), DARE (Yu et al., 2024), and SLERP (Shoemake, 1985). SLERP merges two models using Spherical Linear Interpolation to allow a smoother transition between model parameters while preventing the significant information loss often encountered with direct averaging of model weights. TIES merges models by creating "task vectors" from each model, isolating unique contributions by subtracting an ancestor base model (e.g., Mistral 7B Instruct). These vectors are then averaged with the base model. Its key improvement over previous methods relies on reducing model interference using sparse vectors and a sign consensus method. DARE enhances TIES by reducing delta parameter redundancy, mainly setting them to zero through random pruning and rescaling while maintaining or improving original model performance.
Exploring model merging in the biomedical domain is particularly interesting since merging a general domain model with a domain-specific one could enhance specialized model adaptability and accuracy across a broader range of applications. The objective of this application in the medical domain is not only to improve general-domain capabilities but also to explore the possibility of emergent reasoning and surpassing the performance of baseline models used for merging.

heading: 3.4 Quantization
text: Quantization techniques are pivotal in democratizing LLMs as they enable the execution of LLMs on smaller devices by minimizing memory requirements. In our study, we investigate two core techniques: Activation-aware Weight Quantization (AWQ) and BitsandBytes (BnB). AWQ (Lin et al., 2023) capitalizes on the insight that weights vary in importance, allowing us to skip quantizing critical weights to mitigate performance degradation. Conversely, BnB quantization assigns a fixed precision of 4 or 8 bits to the entire model.

heading: 4 Evaluation Protocol
text: To assess the performance of BioMistral 7B models, we first describe our benchmark of English medical reasoning tasks (Section 4.1) and their multilingual translation (Section 4.2), before presenting the instruction prompting (Section 4.3) and the supervised fine-tuning strategy (Section 4.4) employed for the models’ evaluation.

heading: 4.1 Downstream Tasks
text: To evaluate the performance of the BioMistral 7B model, we selected 10 QA tasks in English from 4 prominent medical corpora covering various specialties, including genetics, anatomy, and clinical cases. These datasets encapsulate real-world scenarios encountered by medical professionals, medical school entrance examination formats, and comprehension tests based on PubMed content. The datasets’ characteristics are provided in Table 1.
MMLU (Hendrycks et al., 2021) comprises exam questions spanning 57 subjects, including philosophy, management, and medical domains. We focused on the 6 subjects relevant to medical and clinical knowledge, and already used to evaluate MedPaLM-2 (Singhal et al., 2023b): college biology, college medicine, anatomy, professional medicine, medical genetics, and clinical knowledge. These subjects were amalgamated to form a consolidated medical-related benchmark, featuring 1,089 questions. As MMLU lacks training data, we fine-tuned our models on MedQA and evaluated their generalization performance on MMLU.
MedQA (Jin et al., 2020) presents questions in the format of the US Medical License Exam (USMLE) and encompasses diverse medical knowledge, including patient profiles, disease symptoms, and drug dosage requirements. The training set comprises 10,178 samples, while the test set contains 1,273 questions. MedQA provides two configurations: four-choice (MedQA) and five-choice (MedQA 5-options) question formats.
MedMCQA (Pal et al., 2022) consists of over 193k MCQA with 4 options each, extracted from Indian medical entrance examinations (AIIMS/NEET). It covers 2,400 healthcare topics across 21 medical subjects. The training set comprises 183k samples and the validation set includes 4,183 questions. Due to the unavailability of answer keys for the test set of 6,150 questions, we adopt a similar approach to Wu et al. (2023), using the validation set for evaluation. For hyperparameter tuning, we perform a random split of the training set into new train/validation splits with 146k and 37k samples each.
PubMedQA (Jin et al., 2019) comprises 211k artificially generated multiple-choice question samples and 1,000 samples labeled by experts. In our evaluation, we adhere to the reasoning-required setting, in which the model has to predict yes, no, or maybe for a given PubMed abstract used as context and a corresponding question. Fine-tuning is performed using 211k artificially labeled samples and performance is accessed on 500 expert-labeled samples during validation and 500 during test as split in BigBio (Fries et al., 2022) and following Chen et al. (2023); Singhal et al. (2023a) protocol.

heading: 4.2 Multilingual Evaluation
text: While the biomedical language models have been extensively evaluated in languages such as English (Lee et al., 2019; Chen et al., 2023), Chinese (Cai et al., 2021; Yang et al., 2023), French (Touchent et al., 2023; Labrak et al., 2023) or Spanish (Carrino et al., 2022), their performance in languages beyond their own remains relatively understudied. This limited multilingual evaluation can be attributed to the scarcity of biomedical tasks available in languages other than English. To address this gap, we conducted a multilingual evaluation using GPT-3.5 Turbo (version 1106) automatic translation via the OpenAI API. We translated our benchmark into 7 languages: Spanish, German, Portuguese, Russian, French, Arabic, and Chinese. Despite the challenges posed by automatic transla-
tion, these tools have shown remarkable improvement in recent years (Neves et al., 2023), enabling cost-effective multilingual evaluation. The methodology for multilingual evaluation and the prompt template are the same as those used in the 3-shot scenario for English. The only differences lie in the translation of the questions, options, and context, while the examples used for few-shot learning remain unchanged.

heading: 4.3 Instruction Prompting
text: All of our instructions adhere to the guidelines outlined for GPT-4’s medical evaluation, as detailed in Nori et al. (2023a). Each task is presented as an MCQA, with answer options associated with letters (A to D or A to E). For a comprehensive list of the instruction prompts, please refer to Appendix F. During inference, the model predicts the next token based on the input prompt, generating probabilities for each token in the vocabulary. To ensure relevance, the vocabulary is filtered to include only tokens (here, choice letters) corresponding to the expected answer options. This approach prevents the model from generating irrelevant tokens or hallucinations (Liang et al., 2023; Beeching et al., 2023; Chen et al., 2023).

heading: 4.4 Supervised Fine-Tuning (SFT)
text: Supervised Fine-Tuning (SFT) is a crucial step involving fine-tuning the model on annotated data to adapt it to specific tasks. To optimize BioMistral’s performance beyond what is achievable with fewshot learning, we conducted SFT on both BioMistral 7B models and the baseline open-source models, using the training sets specified in Table 1. However, traditional SFT methods can be resourceintensive. To address this challenge, we adopted the QLoRa fine-tuning method (Dettmers et al., 2023) and an 8-bit quantization technique (Dettmers et al.,
2022) as more cost-effective alternatives. Additionally, we implemented the improved batching method discussed in Section 3.2 to reduce finetuning time. For detailed hyperparameters used during SFT, please refer to Appendix A.

heading: 5 Results and Discussions
text: In this section, we report, analyze, and discuss the performance of BioMistral 7B models across various dimensions. We begin by examining its performance in a few-shot learning scenario (Section 5.1), followed by an evaluation of the finetuning performances (Section 5.2) of BioMistral 7B compared to several baseline models. The effectiveness of BioMistral 7B model merging strategies is then reported (Section 5.3) before exploring its generalization capabilities across several languages (Section 5.4). Additionally, we analyze the performance of BioMistral quantized versions in a few-shot scenario (Section 5.5). Finally, we delve into its reliability by examining its calibration (Section 5.6) and truthfulness (Section 5.7).

heading: 5.1 Few-shot Learning
text: The few-shot learning evaluation involved applying 3-shot in-context learning based on 3 different sets of randomly selected samples from each dataset’s training set. We limited our samples to 3 due to the model’s 2,048-token context window size. None of the models were fine-tuned on the datasets.
In Table 2, we observe that BioMistral 7B outperforms Mistral 7B Instruct on 8 of the 10 tasks, demonstrating the effectiveness of domain adaptation (Chen et al., 2023; Lee et al., 2019). Additionally, BioMistral 7B surpasses all other opensource biomedical baselines on all tasks in this 3-shot scenario. The observed performances may vary depending on the dataset. For example, on MedQA 4 and 5 options, BioMistral 7B shows a
9.6% and 11.1% increase over MediTron-7B and a 9.0% and 7.0% increase over MedAlpaca 7B, respectively. On MMLU, BioMistral 7B improves performance over previous biomedical LLMs at the 7B scale, with an overall average gain of 6.45% over MedAlpaca 7B, 18.05% over MediTron-7B, and 31.12% over PMC-LLaMA 7B. Similarly, on MedMCQA, BioMistral 7B shows a 10.3% increase over MediTron-7B, 12.7% over MedAlpaca 7B, and 20.4% over PMC-LLaMA 7B. However, in the PubMedQA evaluation, BioMistral’s performance experienced a decline, showing at least a 15.7% lower accuracy compared to other models, likely due to hallucinations caused by imbalanced classes. Overall, GPT-3.5 Turbo remains the best model in this 3-shot scenario.

heading: 5.2 Supervised Fine-Tuning (SFT)
text: We present the performance of BioMistral models and related baselines in Table 3, measured in terms of accuracy. Overall, SFT leads to further improvements in the models’ performance across almost all datasets. Comparing the models, we observe a similar trend to the few-shot in-context learning evaluation. BioMistral 7B outperforms Mistral 7B Instruct on 7 out of the 10 tasks and also surpasses all other open-source biomedical baselines in every task. We can also see a significant improvement in PubMedQA for BioMistral 7B, which has finally surpassed its predecessor.

heading: 5.3 Model Merging
text: As detailed in Section 3.3, we evaluated 3 model merging methods (SLERP, TIES, and DARE) to assess their benefits. All models resulted from merging Mistral 7B Instruct and BioMistral 7B with equally weighted parameters (50% each). Two scenarios are studied: (1) few-shot learning (Table 2),
and (2) supervised fine-tuning (Table 3). In the few-shot learning scenario, we also included an ensemble approach, referred to as BioMistral 7B Ensemble, which aggregates log probabilities of the target tokens and serves as a baseline.
Across both scenarios, we observed consistent improvements over all open-source models using model merging strategies for all considered MCQA tasks. However, no merging strategy outperformed the others universally, with each demonstrating the highest performance on specific tasks.
In the few-shot learning scenario (Table 2), BioMistral 7B Ensemble exhibited a notable increase in accuracy, by 3.7% on College Biology and 30.4% on PubMedQA compared to the standalone BioMistral 7B model. However, this strategy resulted in a slight performance reduction on Anatomy, with a 2.7% drop compared to BioMistral 7B. Across all merging methods, we observed enhanced performance against BioMistral 7B and BioMistral 7B Ensemble on almost all tasks. Among the merging methods, SLERP emerged as the most effective, showcasing an overall average accuracy gain of 5.11% over BioMistral 7B. In contrast, DARE and TIES methods yielded average gains of 4.35% and 0.82%, respectively.
In the context of SFT (Table 3), similar observations were made: model merging methods further enhanced BioMistral’s performance, widening the gap with other open-source biomedical baselines. On average, we observed a gain of 2.06% between the best merged model and BioMistral 7B, and 3.48% compared to Mistral 7B Instruct. Baseline models lagged behind, with a 7.9% overall loss for the best model, MedAlpaca 7B. Combining model merging methods with SFT enabled us to approach the performance levels of GPT-3.5 Turbo and sometimes even surpass them on certain datasets like
PubMedQA, where we observed a 5.14% gain with BioMistral 7B SLERP.

heading: 5.4 Multilingual Generalization
text: We report in Appendix H the detailed few-shot learning performance of all models across the 7 targeted languages. Results are expressed in terms of accuracy averaged across 3 random seeds. Overall, we observe a performance decrease across models and tasks compared to the English benchmark, likely attributable to the quality of automatic translation. Despite this, GPT-3.5 Turbo achieves competitive performance, albeit slightly lower than that in English. We observe that the performance difference between GPT-3.5 Turbo and open-source medical models is similar across languages which could suppose a lack of training data in the targeted language in open-source models and better multilingual capabilities from GPT-3.5 Turbo.
For a given model and task, the performance may vary between languages. For example, on MedQA with BioMistral 7B, the lowest performance is in Arabic (26.3%), while the best is in Spanish (33.7%), representing a delta of 7.4%. Similarly, this trend is observed for GPT-3.5 Turbo with 40.0% accuracy in Chinese and 49.0% in Spanish. Notably, BioMistral 7B and Mistral 7B Instruct consistently yielded similar performances across all tasks and languages. Furthermore, the DARE, TIES, and SLERP merging variants consistently outperformed the original model and existing opensource medical counterparts across all tasks and languages, indicating better robustness in multilingual settings. Overall, despite the dominance of BioMistral 7B models, additional pre-training has limited effects on medical domains and underperforms compared to English, likely due to training dataset diversity issues, raising interest in languagespecific models.

heading: 5.5 Quantization Techniques
text: Table 4 provides an overview of the impact of different quantization techniques on BioMistral performance. Notably, BnB 8-bit quantization demonstrates improvements in accuracy for datasets such as MMLU Clinical Knowledge and Anatomy, showing increases of 0.65% and 1.00%, respectively. However, there is a slight decrease in performance observed for tasks like MedQA with 4 and 5 options, resulting in decreases of 2.61% and 1.06% across all models. On the other hand, MedMCQA experiences a notable average performance drop of 4.05% across all quantization methods, while PubMedQA shows a remarkable 24.1% increase in accuracy when employing the AWQ method.
Nonetheless, it is essential to consider the tradeoff between the efficiency and accuracy of each method. Despite its high compression rate (see Appendix D) and competitive performance, the AWQ + GEMV model exhibits the slowest inference time, taking 421 seconds to process the MMLU professional medicine test set on an RTX 3090. In contrast, the AWQ + GEMM model achieves an 86.23% faster inference time, completing the same task in 57.96 seconds, albeit with a slight performance loss. Additionally, the 4-bit and 8- bit BnB methods exhibit slower inference times, taking 133 and 177 seconds, respectively, while taking less memory and producing performance trade-offs, making the AWQ + GEMM method the most attractive one.

heading: 5.6 Calibration
text: Ensuring model calibration is essential to guarantee that predicted probabilities align with real-world outcomes. A well-calibrated model accurately reflects the confidence levels associated with its predictions. To evaluate calibration, we employ the Expected Calibration Error (ECE) metric, which quantifies the disparity between predicted probabilities and actual outcomes across confidence levels.
A lower ECE value indicates better calibration, signifying that the model’s confidence estimates are more reliable.
ECE =
M∑
m=1
|Bm| n |acc(Bm)− conf(Bm)|
Table 5 presents the calibration and confidence scores for BioMistral 7B and its base model across various languages compared to other open-source medical models. Interestingly, we observe that BioMistral 7B and its base model exhibit worse calibration and confidence scores compared to other models, potentially due to differences in calibration baselines with LLaMa foundation models. Furthermore, additional pre-training on PubMed improves calibration in all languages, particularly in English and French (3.3% ECE gain), with some degradation observed in Chinese (loss of 1.6%). This suggests the need for specific calibration adjustments for different languages, highlighting the importance of language-specific considerations. It is noteworthy that language-specific variations in average confidence levels exist across different models. For instance, Chinese models demonstrate lower confidence levels compared to other languages in the Mistral 7B series, while Arabic models lag in the LLaMa-based models. Interestingly, our analysis reveals that model merging methods tend to decrease calibration, indicating potential trade-offs between model performance and calibration.

heading: 5.7 Truthfulness
text: Truthfulness in language models is essential for preventing the spread of misconceptions and false beliefs. We employ the TruthfulQA benchmark (Lin et al., 2022) to assess truthfulness, which evaluates LLMs’ factual and sensible output across 817 questions and 38 categories, such as finance and politics. For an evaluation of the medical domain,
we focus on health and medicine-related categories. The evaluation consists of two zero-shot prompts: a general assessment prompt and one derived from the MediTron-7B article (see Figure 4).
Table 8 shows that BioMistral 7B outperforms other models across both prompts and demonstrates a 4.0% improvement over GPT-3.5 Turbo.
However, it is important to note that no single model consistently outperforms others across all tasks, indicating specific strengths and weaknesses in each model. Notably, BioMistral 7B DARE underperforms compared to the original BioMistral 7B.
Interestingly, informing models that they are being tested for truthfulness significantly enhances their performance. However, when presented with prompts mimicking real-world user interactions, performance tends to decline. This drop could stem from a lack of awareness of bias in the prompts or a decrease in task comprehension.
Finally, zero-shot prompting poses challenges, particularly for PMC-LLaMA 7B and MediTron7B models, which struggled to provide correct answers in Science and Psychology categories.

heading: 6 Conclusion
text: We introduced BioMistral 7B, a collection of medical LLMs resulting from further pre-training Mistral 7B Instruct on high-quality PubMed Central resources. BioMistral 7B incorporates quantized and merged model variants and demonstrates stateof-the-art performance on the multilingual medical evaluation benchmark compared to other opensource 7B models.
Our future work aims to assess the generation quality of BioMistral 7B through human evaluation. Additionally, we plan to enhance its multilingual and chat capabilities using supervised fine-tuning and direct preference optimization tech-
niques, building on top of experiments conducted by (Rafailov et al., 2023) and Li et al. (2023). Finally, we intend to improve the calibration and reliability of our model by integrating techniques such as Jeffrey’s divergence (Jeffreys, 1946) or Platt scaling (Platt et al., 1999) during the further pretraining process.
Limitations
This study required substantial computational resources, encompassing approximately 5,000 hours of A100 80GB GPU computation. These resources were utilized for model creation, evaluations, experimentation with different architectures, and debugging. Technical issues related to model configurations and performance also necessitated additional computation time. According to documentation from the Jean Zay supercomputer4, the total environmental cost amounted to 1,295,000 Wh or 73.8 kg CO2eq, based on the carbon intensity of the energy grid as reported in the BLOOM environmental cost study conducted on Jean Zay (Luccioni et al., 2022). The valuation of computing hours of the experiments amounts to approximately 3,600 EUR, based on Genci documentation5, or 20,480 USD for AWS on-demand p4d.24xlarge instances. Additionally, the total inference cost for GPT-3.5 Turbo, inherent to the translation and few-shot evaluation process, amounted to 355.47 USD. These costs make reproducing this study challenging when limited financial and material resources are available.
Given the evolving nature of the GPT-3.5 Turbo model, future replication of these experiments may become impractical if the version used is no longer maintained.
While BioMistral 7B is proficient in processing medical terms and concepts close to its training dataset, the model may encounter difficulties with unfamiliar or rare medical procedures or terminology. Furthermore, its reliance on Englishlanguage data results in degraded performance in non-English contexts. It can occasionally cause misinterpretation and lead to erroneous predictions.
Our benchmark offers a framework for academic assessment with selected tasks and metrics, but it might not accurately reflect end users’ actual usage patterns or priorities. Designed for research environments, these criteria may overlook various factors that shape real-world user experiences and preferences.
4http://www.idris.fr/media/jean-zay/jean-zay-consoheure-calcul.pdf
5https://www.edari.fr/documents/Modalitesdacces.pdf
Ethics Statement
Users are solely responsible for the content they generate with BioMistral 7B, and there are no mechanisms in place for addressing harmful, bias, and toxic content disclosure. Any modifications of the models will be released under different version numbers to keep track of the original models related to this paper.
While we introduce BioMistral 7B as a model tailored for the medical domain at large, its evaluation was limited to MCQA datasets, which may not reflect its effectiveness outside this scope. Similar to other LLMs, BioMistral 7B may possess inherent risks and biases that have not yet been thoroughly assessed. Additionally, the model’s performance has not been evaluated in real-world clinical settings. Consequently, we recommend using BioMistral 7B strictly as a research tool and advise against deploying it in production environments for natural language generation or any professional health and medical purposes.
Further evaluation of available language models on various domains is required to assess their capability to generate toxic, rude, or hateful content. To achieve this, the use of datasets such as ToxiGen (Hartvigsen et al., 2022) could provide deeper insights into the subject and help understand how to prevent such behavior. Bias can also significantly impact how language models handle given tasks and may perpetuate stereotypical social biases and demographic attributes observed during training. Two datasets that could be utilized to assess such biases are BOLD (Dhamala et al., 2021) for more general contexts and Discrim-Eval (Tamkin et al., 2023) and SHADR (Guevara et al., 2024), which are specialized for the medical domain.

heading: A Supervised Fine-Tuning Hyperparameters

heading: B Tokenization
text: We adapted the SentencePiece tokenizer (Kudo and Richardson, 2018) of Mistral, which had a vocabulary of 32,000 tokens, by adding a padding token. This padding token is identical to the end-ofsequence token (
).

heading: C Grouping method algorithm
text: Algorithm 1: Pseudocode of the grouping method.
Data: Input list of unequal length sequences of token Result: A list of 2048 token long sequences separator ← 
; tokens← flatten(sequences, separator); length← size(tokens); if length >= 2048 then
length← (length//2048)× 2048; for i← 2048 to length do
result← tokens[i : i+ 2048]; end
else result← tokens; end

heading: D Memory Footprint

heading: E Training Loss
text: As described in section 3.1, one of our pretraining strategies was to achieve the 1.5-epoch milestone, similar to the Zephyr model. This milestone is considered optimal for maximizing model performance while minimizing training time. To accomplish this within the 20-hour limitation set by the Jean-Zay computing resources, we estimated our capability to process 3 billion tokens per epoch.
Figure 1 shows our training loss during the further pre-training of Mistral 7B Instruct v0.1 on PubMed Central. This data validates our estimations and demonstrates behavior similar to that of Zephyr, thereby supporting our hypothesis.
F Instruction template for multiple choice question answering
Figure 2 display the instruction template used for all datasets. In the case of PubMedQA, the prompt includes a context before the question and the three answer option: yes, no, or maybe are formulated as a multiple choice question, where A is yes, B is no, and C is maybe, matching the testing method done by Liévin et al. (2023).

heading: G TruthfulQA

heading: H Multilingual Results

heading: Future Work
text: Our future work aims to assess the generation quality of BioMistral 7B through human evaluation. Additionally, we plan to enhance its multilingual and chat capabilities using supervised fine-tuning and direct preference optimization tech-
niques, building on top of experiments conducted by (Rafailov et al. , 2023) and Li et al. (2023). Finally, we intend to improve the calibration and reliability of our model by integrating techniques such as Jeffrey’s divergence (Jeffreys, 1946) or Platt scaling (Platt et al. , 1999) during the further pretraining process. Limitations
This study required substantial computational resources, encompassing approximately 5,000 hours of A100 80GB GPU computation. These resources were utilized for model creation, evaluations, experimentation with different architectures, and debugging. Technical issues related to model configurations and performance also necessitated additional computation time. According to documentation from the Jean Zay supercomputer4, the total environmental cost amounted to 1,295,000 Wh or 73. 8 kg CO2eq, based on the carbon intensity of the energy grid as reported in the BLOOM environmental cost study conducted on Jean Zay (Luccioni et al. , 2022). The valuation of computing hours of the experiments amounts to approximately 3,600 EUR, based on Genci documentation5, or 20,480 USD for AWS on-demand p4d. 24xlarge instances. Additionally, the total inference cost for GPT-3. 5 Turbo, inherent to the translation and few-shot evaluation process, amounted to 355. 47 USD. These costs make reproducing this study challenging when limited financial and material resources are available. Given the evolving nature of the GPT-3. 5 Turbo model, future replication of these experiments may become impractical if the version used is no longer maintained. While BioMistral 7B is proficient in processing medical terms and concepts close to its training dataset, the model may encounter difficulties with unfamiliar or rare medical procedures or terminology. Furthermore, its reliance on Englishlanguage data results in degraded performance in non-English contexts. It can occasionally cause misinterpretation and lead to erroneous predictions. Our benchmark offers a framework for academic assessment with selected tasks and metrics, but it might not accurately reflect end users’ actual usage patterns or priorities. Designed for research environments, these criteria may overlook various factors that shape real-world user experiences and preferences. 4http://www. idris. fr/media/jean-zay/jean-zay-consoheure-calcul. pdf
5https://www. edari. fr/documents/Modalitesdacces