|
--- |
|
license: apache-2.0 |
|
tags: |
|
- medical |
|
datasets: |
|
- biomed |
|
--- |
|
# BioMedGPT-LM-7B |
|
|
|
In this repo, we present a medical language model named BioMedGPT-LM which is the first commercial-friendly GPT model in the biomedical domain and has demonstrated |
|
superior performance over existing LLMs of the same parameter size. We are releasing a 7B model **BioMedGPT-LM-7B** which is LLaMA2-7b-chat finetuned on the PMC abstracts and papers from the S2ORC. |
|
|
|
|
|
|
|
|
|
### Training Details |
|
|
|
|
|
|
|
The model was trained with the following hyperparameters: |
|
|
|
* Epochs: 5 |
|
* Batch size: 192 |
|
* Cutoff length: 2048 |
|
* Learning rate: 2e-5 |
|
|
|
Overview BioMedGPT-LM-7B was finetuned on over 26 billion tokens highly pertinent to the field of biomedicine. The fine-tuning data are extracted from 5.5 million biomedical papers in S2ORC data using PubMed Central |
|
(PMC)-ID and PubMed ID as criteria. |
|
|
|
|
|
### Model Developers |
|
PharMolix |
|
|
|
### How to Use |
|
BioMedGPT-LM-7B is a part of **[BioMedGPT-10B](https://github.com/BioFM/OpenBioMed)**, an open-source version of BioMedGPT. BioMedGPT is a multimodal generative pre-trained transformer (GPT) for biomedicine, which bridges the natural language modality and diverse biomed- |
|
ical data modalities via a single GPT model. BioMedGPT aligns different biological modalities with the text modality via BioMedGPT-LM. The details of BioMedGPT-10B and BioMedGPT-LM-7B can be found in the [technical report](). |
|
 |
|
|
|
|
|
|
|
|
|
**Intended Use Cases** |
|
|
|
| **Method** | Parameters (B) | Setting | MedMCQA(\%) | PubMedQA(\%) | |
|
|------------------------|----------------|-----------|-------------|--------------| |
|
| Human (pass)* | - | Manual | - | 60.0 | |
|
| Human (expert)* | - | Manual | 90 | 78.0 | |
|
|------------------------|----------------|-----------|-------------|--------------| |
|
| InstructGPT* | 175 | zero-shot | 44.0 | 73.2 | |
|
| ChatGPT* | - | zero-shot | 44.7 | 63.9 | |
|
| Llama* | 7 | zero-shot | 24.3 | 5.2 | |
|
| Llama2 | 7 | zero-shot | 30.6 | 3.7 | |
|
| Llama2-Chat | 7 | zero-shot | 35.5 | 21.9 | |
|
|------------------------|----------------| --------- |-------------|--------------| |
|
| Llama | 7 |Fine-tuing | 48.2 | 73.4 | |
|
| Llama2-Chat | 7 |Fine-tuing | 48.3 | 75.5 | |
|
| PMC-Llama | 7 |Fine-tuing | 50.5 | 69.5 | |
|
|------------------------|----------------|-----------|-------------|--------------| |
|
| BioMed-Llama2 | 7 |Fine-tuing | **51.4** | **76.1** | |
|
|
|
**Out-of-scope Uses** |
|
|
|
|
|
### Technical Report |
|
"BioMedGPT: Open Multimodal Generative Pre-trained Transformer for BioMedicine" |
|
|
|
|
|
### github |
|
[https://github.com/BioFM/OpenBioMed](https://github.com/BioFM/OpenBioMed) |
|
|
|
|
|
### Limitations |
|
|
|
[Highlight any limitations or potential issues of your model.] |
|
|
|
|
|
|
