|
--- |
|
license: cc-by-nc-4.0 |
|
datasets: |
|
- argilla/dpo-mix-7k |
|
- nvidia/HelpSteer |
|
- jondurbin/airoboros-3.2 |
|
- hkust-nlp/deita-10k-v0 |
|
- LDJnr/Capybara |
|
- HPAI-BSC/CareQA |
|
- GBaker/MedQA-USMLE-4-options |
|
- lukaemon/mmlu |
|
- bigbio/pubmed_qa |
|
- openlifescienceai/medmcqa |
|
- bigbio/med_qa |
|
- HPAI-BSC/better-safe-than-sorry |
|
- HPAI-BSC/pubmedqa-cot |
|
- HPAI-BSC/medmcqa-cot |
|
- HPAI-BSC/medqa-cot |
|
language: |
|
- en |
|
library_name: transformers |
|
tags: |
|
- biology |
|
- medical |
|
pipeline_tag: question-answering |
|
--- |
|
# Aloe: A New Family of Healthcare LLMs |
|
|
|
Aloe is a new family of healthcare LLMs that is highly competitive with all previous open models of its range and reaches state-of-the-art results at its size by using model merging and advanced prompting strategies. Aloe scores high in metrics measuring ethics and factuality, thanks to a combined red teaming and alignment effort. Complete training details, model merging configurations, and all training data (including synthetically generated data) will be shared. Additionally, the prompting repository used in this work to produce state-of-the-art results during inference will also be shared. Aloe comes with a healthcare-specific risk assessment to contribute to the safe use and deployment of such systems. |
|
|
|
<img src="https://cdn-uploads.huggingface.co/production/uploads/62972c4979f193515da1d38e/xlssx5_3_kLQlJlmE-aya.png" width="95%"> |
|
|
|
## Model Details |
|
|
|
### [](https://huggingface.co/templates/model-card-example#model-description)Model Description |
|
|
|
- **Developed by:**聽[HPAI](https://hpai.bsc.es/) |
|
- **Model type:**聽Causal decoder-only transformer language model |
|
- **Language(s) (NLP):**聽English (mainly) |
|
- **License:**聽This model is based on Meta Llama 3 8B and is governed by the [Meta Llama 3 License](https://llama.meta.com/llama3/license/). All our modifications are available with a [CC BY-NC 4.0](https://creativecommons.org/licenses/by-nc/4.0/) license. |
|
- **Finetuned from model :** [meta-llama/Meta-Llama-3-8B](https://huggingface.co/meta-llama/Meta-Llama-3-8B) |
|
|
|
### [](https://huggingface.co/templates/model-card-example#model-sources-optional)Model Sources [optional] |
|
|
|
- **Repository:**聽https://github.com/HPAI-BSC/prompt_engine (more coming soon) |
|
- **Paper:** https://arxiv.org/abs/2405.01886 (more coming soon) |
|
|
|
## Model Performance |
|
|
|
Aloe has been tested on the most popular healthcare QA datasets, with and without medprompting inference technique. Results show competitive performance, even against bigger models. |
|
|
|
<img src="https://cdn-uploads.huggingface.co/production/uploads/62f7a16192950415b637e201/rQ4z-qXzKN44oAcFDbHi2.png" width="95%"> |
|
|
|
Results using advanced prompting methods (aka Medprompt) are achieved through a [repo](https://github.com/HPAI-BSC/prompt_engine) made public with this work. |
|
|
|
## Uses |
|
|
|
### Direct Use |
|
|
|
We encourage the use of Aloe for research purposes, as a stepping stone to build better foundational models for healthcare. |
|
|
|
### Out-of-Scope Use |
|
|
|
These models are not to be used for clinical practice, medical diagnosis, or any other form of direct or indirect healthcare advice. Models are prone to error and can produce toxic content. The use of Aloe models for activities harmful for individuals, such as spam, fraud, or impersonation, is prohibited. |
|
|
|
## Bias, Risks, and Limitations |
|
|
|
We consider three risk cases: |
|
- Healthcare professional impersonation, a fraudulent behaviour which currently generates billions of dollars in [profit](https://www.justice.gov/opa/pr/justice-department-charges-dozens-12-billion-health-care-fraud). A model such as Aloe could be used to increase the efficacy of such deceiving activities, making them more widespread. The main preventive actions are public literacy on the unreliability of digitised information and the importance of medical registration, and legislation enforcing AI-generated content disclaimers. |
|
- Medical decision-making without professional supervision. While this is already an issue in modern societies (eg self-medication) a model such as Aloe, capable of producing high-quality conversational data, can facilitate self-delusion, particularly in the presence of sycophancy. By producing tailored responses, it can also be used to generate actionable answers. Public literacy on the dangers of self-diagnosis is one of the main defences, together with the introduction of disclaimers and warnings on the models' outputs. |
|
- Access to information on dangerous substances or procedures. While the literature on sensitive content can already be found on different sources (eg libraries, internet, dark web), LLMs can centralize such access, making it nearly impossible to control the flow of such information. Model alignment can help in that regard, but so far the effects remain insufficient, as jailbreaking methods still overcome it. |
|
|
|
Table below shows the performance of Aloe at several AI safety tasks: |
|
|
|
<img src="https://cdn-uploads.huggingface.co/production/uploads/62972c4979f193515da1d38e/T6Jblpf1kmTkM04K716rM.png" width="95%"> |
|
|
|
### Recommendations |
|
|
|
We avoid the use of all personal data in our training. Model safety cannot be guaranteed. Aloe can produce toxic content under the appropriate prompts. For these reasons, minors should not be left alone to interact with Aloe without supervision. |
|
|
|
## How to Get Started with the Model |
|
|
|
Use the code below to get started with the model. You can run conversational inference using the Transformers pipeline abstraction, or by leveraging the Auto classes with the `generate()` function. Let's see examples of both. |
|
|
|
#### Transformers pipeline |
|
|
|
```python |
|
import transformers |
|
import torch |
|
|
|
model_id = "HPAI-BSC/Llama3-Aloe-8B-Alpha" |
|
|
|
pipeline = transformers.pipeline( |
|
"text-generation", |
|
model=model_id, |
|
model_kwargs={"torch_dtype": torch.bfloat16}, |
|
device_map="auto", |
|
) |
|
|
|
messages = [ |
|
{"role": "system", "content": "You are an expert medical assistant named Aloe, developed by the High Performance Artificial Intelligence Group at Barcelona Supercomputing Center(BSC). You are to be a helpful, respectful, and honest assistant."}, |
|
{"role": "user", "content": "Hello."}, |
|
] |
|
|
|
prompt = pipeline.tokenizer.apply_chat_template( |
|
messages, |
|
tokenize=False, |
|
add_generation_prompt=True |
|
) |
|
|
|
terminators = [ |
|
pipeline.tokenizer.eos_token_id, |
|
pipeline.tokenizer.convert_tokens_to_ids("<|eot_id|>") |
|
] |
|
|
|
outputs = pipeline( |
|
prompt, |
|
max_new_tokens=256, |
|
eos_token_id=terminators, |
|
do_sample=True, |
|
temperature=0.6, |
|
top_p=0.9, |
|
) |
|
print(outputs[0]["generated_text"][len(prompt):]) |
|
``` |
|
|
|
#### Transformers AutoModelForCausalLM |
|
|
|
```python |
|
from transformers import AutoTokenizer, AutoModelForCausalLM |
|
import torch |
|
|
|
model_id = "HPAI-BSC/Llama3-Aloe-8B-Alpha" |
|
|
|
tokenizer = AutoTokenizer.from_pretrained(model_id) |
|
model = AutoModelForCausalLM.from_pretrained( |
|
model_id, |
|
torch_dtype=torch.bfloat16, |
|
device_map="auto", |
|
) |
|
|
|
messages = [ |
|
{"role": "system", "content": "You are an expert medical assistant named Aloe, developed by the High Performance Artificial Intelligence Group at Barcelona Supercomputing Center(BSC). You are to be a helpful, respectful, and honest assistant."}, |
|
{"role": "user", "content": "Hello"}, |
|
] |
|
|
|
input_ids = tokenizer.apply_chat_template( |
|
messages, |
|
add_generation_prompt=True, |
|
return_tensors="pt" |
|
).to(model.device) |
|
|
|
terminators = [ |
|
tokenizer.eos_token_id, |
|
tokenizer.convert_tokens_to_ids("<|eot_id|>") |
|
] |
|
|
|
outputs = model.generate( |
|
input_ids, |
|
max_new_tokens=256, |
|
eos_token_id=terminators, |
|
do_sample=True, |
|
temperature=0.6, |
|
top_p=0.9, |
|
) |
|
response = outputs[0][input_ids.shape[-1]:] |
|
print(tokenizer.decode(response, skip_special_tokens=True)) |
|
``` |
|
|
|
## Training Details |
|
|
|
Supervised fine-tuning on top of Llama 3 8B using medical and general domain datasets, model merging using DARE-TIES process, two-stage DPO process for human preference alignment. More details coming soon. |
|
|
|
### Training Data |
|
|
|
- Medical domain datasets, including synthetic data generated using Mixtral-8x7B and Genstruct |
|
- HPAI-BSC/pubmedqa-cot |
|
- HPAI-BSC/medqa-cot |
|
- HPAI-BSC/medmcqa-cot |
|
- LDJnr/Capybara |
|
- hkust-nlp/deita-10k-v0 |
|
- jondurbin/airoboros-3.2 |
|
- argilla/dpo-mix-7k |
|
- nvidia/HelpSteer |
|
- Custom preference data with adversarial prompts generated from Anthropic Harmless, Chen et al., and original prompts |
|
|
|
## Evaluation |
|
|
|
### Testing Data, Factors & Metrics |
|
|
|
#### Testing Data |
|
|
|
- [MedQA (USMLE)](https://huggingface.co/datasets/bigbio/med_qa) |
|
- [MedMCQA](https://huggingface.co/datasets/medmcqa) |
|
- [PubMedQA](https://huggingface.co/datasets/bigbio/pubmed_qa) |
|
- [MMLU-Medical](https://huggingface.co/datasets/lukaemon/mmlu) |
|
- [MedQA-4-Option](https://huggingface.co/datasets/GBaker/MedQA-USMLE-4-options) |
|
- [CareQA](https://huggingface.co/datasets/HPAI-BSC/CareQA) |
|
|
|
#### Metrics |
|
|
|
- Accuracy: suite the evaluation of multiple-choice question-answering tasks. |
|
|
|
### Results |
|
|
|
<img src="https://cdn-uploads.huggingface.co/production/uploads/62972c4979f193515da1d38e/STlPSggXr9P9JeWAvmAsi.png" width="90%"> |
|
|
|
#### Summary |
|
|
|
To compare Aloe with the most competitive open models (both general purpose and healthcare-specific) we use popular healthcare datasets (PubMedQA, MedMCQA, MedQA and MMLU for six medical tasks only), together with the new and highly reliable CareQA. We produce the standard MultiMedQA score for reference, by computing the weighted average accuracy on all scores except CareQA. Additionally, we calculate the arithmetic mean across all datasets. The Medical MMLU is calculated by averaging the six medical subtasks: Anatomy, Clinical knowledge, College Biology, College medicine, Medical genetics, and Professional medicine. |
|
|
|
Benchmark results indicate the training conducted on Aloe has boosted its performance above Llama3-8B-Instruct. Llama3-Aloe-8B-Alpha outperforms larger models like Meditron 70B, and is close to larger base models, like Yi-34. For the former, this gain is consistent even when using SC-CoT, using their best-reported variant. All these results make Llama3-Aloe-8B-Alpha the best healthcare LLM of its size. |
|
|
|
With the help of prompting techniques the performance of Llama3-Aloe-8B-Alpha is significantly improved. Medprompting in particular provides a 7% increase in reported accuracy, after which Llama3-Aloe-8B-Alpha only lags behind the ten times bigger Llama-3-70B-Instruct. This improvement is mostly consistent across medical fields. Llama3-Aloe-8B-Alpha with medprompting beats the performance of Meditron 70B with their self reported 20 shot SC-CoT in MMLU med and is slightly worse in the other benchmarks. |
|
|
|
## Environmental Impact |
|
|
|
- **Hardware Type:**聽4xH100 |
|
- **Hours used:**聽7,000 |
|
- **Hardware Provider:**聽Barcelona Supercomputing Center |
|
- **Compute Region:**聽Spain |
|
- **Carbon Emitted:**聽439.25kg |
|
|
|
## Model Card Authors |
|
[Ashwin Kumar Gururajan](https://huggingface.co/G-AshwinKumar) |
|
|
|
## Model Card Contact |
|
|
|
mailto:hpai@bsc.es |
|
|
|
## Citations |
|
|
|
If you use this repository in a published work, please cite the following papers as source: |
|
|
|
``` |
|
@misc{gururajan2024aloe, |
|
title={Aloe: A Family of Fine-tuned Open Healthcare LLMs}, |
|
author={Ashwin Kumar Gururajan and Enrique Lopez-Cuena and Jordi Bayarri-Planas and Adrian Tormos and Daniel Hinjos and Pablo Bernabeu-Perez and Anna Arias-Duart and Pablo Agustin Martin-Torres and Lucia Urcelay-Ganzabal and Marta Gonzalez-Mallo and Sergio Alvarez-Napagao and Eduard Ayguad茅-Parra and Ulises Cort茅s Dario Garcia-Gasulla}, |
|
year={2024}, |
|
eprint={2405.01886}, |
|
archivePrefix={arXiv}, |
|
primaryClass={cs.CL} |
|
} |
|
``` |
|
|
