---
license: apache-2.0
datasets:
- Henrychur/MMedC
language:
- en
- zh
- ja
- fr
- ru
- es
tags:
- medical
---
# MMedLM
[💻Github Repo](https://github.com/MAGIC-AI4Med/MMedLM)   [🖨️arXiv Paper](https://arxiv.org/abs/2402.13963)

The official model weights for "Towards Building Multilingual Language Model for Medicine".

[MMedLM 2](https://huggingface.co/Henrychur/MMedLM2) has been released now. MMedLM2 is a more powerful multilingual medical foundation model, which has undergone the same medical data enhancement pipeline as MMedLM.

## Introduction
This repo contains MMedLM, a multilingual medical foundation model with 7 billion parameters. MMedLM builds upon the foundation of InternLM and has been further pretrained on MMedC, a comprehensive multilingual medical corpus. This further pretraining enhances the model's medical-domain knowledge.

The model underwent further pretraining on MMedC with the following hyperparameters:
- Iterations: 15000
- Global batch size: 512
- Cutoff length: 2048
- Learning rate: 2e-5


The model can be loaded as follows:
```py
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
tokenizer = AutoTokenizer.from_pretrained("Henrychur/MMedLM", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("Henrychur/MMedLM", torch_dtype=torch.float16, trust_remote_code=True)
```

-  Note that this is a foundation model that has not undergone instruction fine-tuning.
-  Testing has found that using the latest version of transformers will result in errors. It is recommended to use transformers==4.28.1.
## News
[2024.2.21] Our pre-print paper is released ArXiv. Dive into our findings [here](https://arxiv.org/abs/2402.13963).

[2024.2.20] We release [MMedLM](https://huggingface.co/Henrychur/MMedLM) and [MMedLM 2](https://huggingface.co/Henrychur/MMedLM2). With an auto-regressive continues training on MMedC, these models achieves superior performance compared to all other open-source models, even rivaling GPT-4 on MMedBench.

[2023.2.20] We release [MMedC](https://huggingface.co/datasets/Henrychur/MMedC), a multilingual medical corpus containing 25.5B tokens.

[2023.2.20] We release [MMedBench](https://huggingface.co/datasets/Henrychur/MMedBench), a new multilingual medical multi-choice question-answering
benchmark with rationale. Check out the leaderboard [here](https://henrychur.github.io/MultilingualMedQA/).

## Evaluation on MMedBench
The further pretrained MMedLM 2 showcast it's great performance in medical domain across different language.

| Method           | Size | Year    | MMedC     | MMedBench | English        | Chinese        | Japanese       | French         | Russian        | Spanish        | Avg.           |
|------------------|------|---------|-----------|-----------|----------------|----------------|----------------|----------------|----------------|----------------|----------------|
| GPT-3.5          | -    | 2022.12 | ✗ | ✗ | 56.88          | 52.29          | 34.63          | 32.48          | 66.36          | 66.06          | 51.47          |
| GPT-4            | -    | 2023.3  | ✗ | ✗ | 78.00 | 75.07 | 72.91 | 56.59 | 83.62 | 85.67 | 74.27 |
| Gemini-1.0 pro   | -    | 2024.1  | ✗ | ✗ | 53.73          | 60.19          | 44.22          | 29.90          | 73.44          | 69.69          | 55.20          |
| BLOOMZ           | 7B   | 2023.5  | ✗ | trainset  | 43.28          | 58.06          | 32.66          | 26.37          | 62.89          | 47.34          | 45.10          |
| InternLM         | 7B   | 2023.7  | ✗ | trainset  | 44.07          | 64.62          | 37.19          | 24.92          | 58.20          | 44.97          | 45.67          |
| Llama\ 2         | 7B   | 2023.7  | ✗ | trainset  | 43.36          | 50.29          | 25.13          | 20.90          | 66.80          | 47.10          | 42.26          |
| MedAlpaca        | 7B   | 2023.3  | ✗ | trainset  | 46.74          | 44.80          | 29.64          | 21.06          | 59.38          | 45.00          | 41.11          |
| ChatDoctor       | 7B   | 2023.4  | ✗ | trainset  | 43.52          | 43.26          | 25.63          | 18.81          | 62.50          | 43.44          | 39.53          |
| PMC-LLaMA        | 7B   | 2023.4  | ✗ | trainset  | 47.53          | 42.44          | 24.12          | 20.74          | 62.11          | 43.29          | 40.04          |
| Mistral          | 7B   | 2023.10 | ✗ | trainset  | 61.74 | 71.10          | 44.72          | 48.71          | 74.22          | 63.86          | 60.73          |
| InternLM\ 2      | 7B   | 2024.2  | ✗ | trainset  | 57.27          | 77.55          | 47.74          | 41.00          | 68.36          | 59.59          | 58.59          |
| MMedLM~(Ours)    | 7B   | -       | ✗ | trainset  | 49.88          | 70.49          | 46.23          | 36.66          | 72.27          | 54.52          | 55.01          |
| MMedLM\ 2~(Ours) | 7B   | -       | ✗ | trainset  | 61.74 | 80.01 | 61.81 | 52.09 | 80.47 | 67.65 | 67.30 |
- GPT and Gemini is evluated under zero-shot setting through API
- Open-source models first undergo training on the trainset of MMedBench before evaluate. 

## Contact
If you have any question, please feel free to contact qiupengcheng@pjlab.org.cn.

## Citation
```
@misc{qiu2024building,
      title={Towards Building Multilingual Language Model for Medicine}, 
      author={Pengcheng Qiu and Chaoyi Wu and Xiaoman Zhang and Weixiong Lin and Haicheng Wang and Ya Zhang and Yanfeng Wang and Weidi Xie},
      year={2024},
      eprint={2402.13963},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}
```