Henrychur commited on
Commit
7ec96fa
1 Parent(s): b3f5e44

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +85 -0
README.md CHANGED
@@ -1,3 +1,88 @@
1
  ---
2
  license: cc-by-4.0
 
 
 
 
 
 
 
 
 
 
 
3
  ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  ---
2
  license: cc-by-4.0
3
+ datasets:
4
+ - Henrychur/MMedC
5
+ language:
6
+ - en
7
+ - zh
8
+ - ja
9
+ - fr
10
+ - ru
11
+ - es
12
+ tags:
13
+ - medical
14
  ---
15
+ # MMedLM
16
+ [💻Github Repo](https://github.com/MAGIC-AI4Med/MMedLM) [🖨️arXiv Paper](https://arxiv.org/abs/2402.13963)
17
+
18
+ The official model weights for "Towards Building Multilingual Language Model for Medicine".
19
+
20
+
21
+ ## Introduction
22
+ This repo contains MMedLM 2-1.8B , a multilingual medical foundation model with 1.8 billion parameters. MMedLM 2-1.8B builds upon the foundation of InternLM 2-1.8B and has been further pretrained on MMedC, a comprehensive multilingual medical corpus. This further pretraining enhances the model's medical-domain knowledge.
23
+ With an auto-regressive continues training on MMedC, MMedLM 2-1.8B can exceed the performance of most 7B models, including InternLM and LLaMA 2.
24
+
25
+ The model underwent further pretraining on MMedC with the following hyperparameters:
26
+ - Iterations: 15000
27
+ - Global batch size: 512
28
+ - Cutoff length: 2048
29
+ - Learning rate: 2e-5
30
+
31
+ The model can be loaded as follows:
32
+ ```py
33
+ import torch
34
+ from transformers import AutoTokenizer, AutoModelForCausalLM
35
+ tokenizer = AutoTokenizer.from_pretrained("Henrychur/MMedLM2-1.8B", trust_remote_code=True)
36
+ model = AutoModelForCausalLM.from_pretrained("Henrychur/MMedLM2-1.8B", torch_dtype=torch.float16, trust_remote_code=True)
37
+ ```
38
+
39
+ - Note that this is a foundation model that has not undergone instruction fine-tuning.
40
+ ## News
41
+ [2023.3.1] We release [MMedLM 2-1.8B](https://huggingface.co/Henrychur/MMedLM2-1.8B), a 1.8B light-weight model based on InternLM 2-1.8B. With an auto-regressive continues training on MMedC, MMedLM 2-1.8B can exceed the performance of most 7B models, including InternLM and LLaMA 2.
42
+
43
+ [2024.2.21] Our pre-print paper is released ArXiv. Dive into our findings [here](https://arxiv.org/abs/2402.13963).
44
+
45
+ [2024.2.20] We release [MMedLM](https://huggingface.co/Henrychur/MMedLM) and [MMedLM 2](https://huggingface.co/Henrychur/MMedLM2). With an auto-regressive continues training on MMedC, these models achieves superior performance compared to all other open-source models, even rivaling GPT-4 on MMedBench.
46
+
47
+ [2023.2.20] We release [MMedC](https://huggingface.co/datasets/Henrychur/MMedC), a multilingual medical corpus containing 25.5B tokens.
48
+
49
+ [2023.2.20] We release [MMedBench](https://huggingface.co/datasets/Henrychur/MMedBench), a new multilingual medical multi-choice question-answering
50
+ benchmark with rationale. Check out the leaderboard [here](https://henrychur.github.io/MultilingualMedQA/).
51
+
52
+ ## Evaluation on MMedBench
53
+ The further pretrained MMedLM 2 showcast it's great performance in medical domain across different language.
54
+
55
+ | Method | Size | Year | MMedC | MMedBench | English | Chinese | Japanese | French | Russian | Spanish | Avg. |
56
+ |------------------|------|---------|-----------|-----------|----------------|----------------|----------------|----------------|----------------|----------------|----------------|
57
+ | GPT-3.5 | - | 2022.12 | ✗ | ✗ | 56.88 | 52.29 | 34.63 | 32.48 | 66.36 | 66.06 | 51.47 |
58
+ | GPT-4 | - | 2023.3 | ✗ | ✗ | 78.00 | 75.07 | 72.91 | 56.59 | 83.62 | 85.67 | 74.27 |
59
+ | Gemini-1.0 pro | - | 2024.1 | ✗ | ✗ | 53.73 | 60.19 | 44.22 | 29.90 | 73.44 | 69.69 | 55.20 |
60
+ | BLOOMZ | 7B | 2023.5 | ✗ | trainset | 43.28 | 58.06 | 32.66 | 26.37 | 62.89 | 47.34 | 45.10 |
61
+ | InternLM | 7B | 2023.7 | ✗ | trainset | 44.07 | 64.62 | 37.19 | 24.92 | 58.20 | 44.97 | 45.67 |
62
+ | Llama 2 | 7B | 2023.7 | ✗ | trainset | 43.36 | 50.29 | 25.13 | 20.90 | 66.80 | 47.10 | 42.26 |
63
+ | MedAlpaca | 7B | 2023.3 | ✗ | trainset | 46.74 | 44.80 | 29.64 | 21.06 | 59.38 | 45.00 | 41.11 |
64
+ | ChatDoctor | 7B | 2023.4 | ✗ | trainset | 43.52 | 43.26 | 25.63 | 18.81 | 62.50 | 43.44 | 39.53 |
65
+ | PMC-LLaMA | 7B | 2023.4 | ✗ | trainset | 47.53 | 42.44 | 24.12 | 20.74 | 62.11 | 43.29 | 40.04 |
66
+ | Mistral | 7B | 2023.10 | ✗ | trainset | 61.74 | 71.10 | 44.72 | 48.71 | 74.22 | 63.86 | 60.73 |
67
+ | InternLM 2 | 1.8B | 2024.2 | ✗ | trainset |38.49 |64.1 |32.16|18.01|53.91|36.83|40.58|
68
+ | InternLM 2 | 7B | 2024.2 | ✗ | trainset | 57.27 | 77.55 | 47.74 | 41.00 | 68.36 | 59.59 | 58.59 |
69
+ | MMedLM (Ours) | 7B | - | ✓ | trainset | 49.88 | 70.49 | 46.23 | 36.66 | 72.27 | 54.52 | 55.01 |
70
+ | MMedLM 2(Ours) | 7B | - | ✓ | trainset | 61.74 | 80.01 | 61.81 | 52.09 | 80.47 | 67.65 | 67.30 |
71
+ | MMedLM 2(Ours) | 1.8B | - | ✓ | trainset | 45.40 | 66.78 | 42.21 | 25.56 | 69.14 | 43.40 | 48.75 |
72
+ - GPT and Gemini is evluated under zero-shot setting through API
73
+ - Open-source models first undergo training on the trainset of MMedBench before evaluate.
74
+
75
+ ## Contact
76
+ If you have any question, please feel free to contact qiupengcheng@pjlab.org.cn.
77
+
78
+ ## Citation
79
+ ```
80
+ @misc{qiu2024building,
81
+ title={Towards Building Multilingual Language Model for Medicine},
82
+ author={Pengcheng Qiu and Chaoyi Wu and Xiaoman Zhang and Weixiong Lin and Haicheng Wang and Ya Zhang and Yanfeng Wang and Weidi Xie},
83
+ year={2024},
84
+ eprint={2402.13963},
85
+ archivePrefix={arXiv},
86
+ primaryClass={cs.CL}
87
+ }
88
+ ```