README.md

---
license: apache-2.0
language:
- en
tags:
- medical
- palmyra
---
# Palmyra-med-20b

## Model description
**Palmyra-Med-20b** is a 20 billion parameter Large Language Model that has been uptrained on 
**Palmyra-Large** with a specialized custom-curated medical dataset. 
The main objective of this model is to enhance performance in tasks related to medical dialogue
and question-answering.

- **Developed by:** [https://writer.com/](https://writer.com/);
- **Model type:** Causal decoder-only;
- **Language(s) (NLP):** English;
- **License:** Apache 2.0;
- **Finetuned from model:** [Palmyra-Large](https://huggingface.co/Writer/palmyra-large).

### Model Source

[Palmyra-Med: Instruction-Based Fine-Tuning of LLMs Enhancing Medical Domain Performance](https://dev.writer.com/docs/palmyra-med-instruction-based-fine-tuning-of-llms-enhancing-medical-domain-performance)


## Uses


### Out-of-Scope Use

Production use without adequate assessment of risks and mitigation; any use cases which may be considered irresponsible or harmful. 

## Bias, Risks, and Limitations

Palmyra-Med-20B is mostly trained on English data, and will not generalize appropriately to other languages. Furthermore, as it is trained on a large-scale corpora representative of the web, it will carry the stereotypes and biases commonly encountered online.

### Recommendations

We recommend users of Palmyra-Med-20B to develop guardrails and to take appropriate precautions for any production use.


## Usage 
The model is compatible with the huggingface `AutoModelForCausalLM` and can be easily run on a single 40GB A100.

```py
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

model_name = "Writer/palmyra-med-20b"

tokenizer = AutoTokenizer.from_pretrained(model_name, use_fast=False)

model = AutoModelForCausalLM.from_pretrained(
    model_name,
    device_map="auto",
    torch_dtype=torch.float16,
)

prompt = "Can you explain in simple terms how vaccines help our body fight diseases?"

input_text = (
    "A chat between a curious user and an artificial intelligence assistant. "
    "The assistant gives helpful, detailed, and polite answers to the user's questions. "
    "USER: {prompt} "
    "ASSISTANT:"
)

model_inputs = tokenizer(input_text.format(prompt=prompt), return_tensors="pt").to(
    "cuda"
)

gen_conf = {
    "temperature": 0.7,
    "repetition_penalty": 1.0,
    "max_new_tokens": 512,
    "do_sample": True,
}

out_tokens = model.generate(**model_inputs, **gen_conf)

response_ids = out_tokens[0][len(model_inputs.input_ids[0]) :]
output = tokenizer.decode(response_ids, skip_special_tokens=True)

print(output)
## output ##
# Vaccines stimulate the production of antibodies by the body's immune system.
# Antibodies are proteins produced by B lymphocytes in response to foreign substances,such as viruses and bacteria.
# The antibodies produced by the immune system can bind to and neutralize the pathogens, preventing them from invading and damaging the host cells.
# Vaccines work by introducing antigens, which are components of the pathogen, into the body.
# The immune system then produces antibodies against the antigens, which can recognize and neutralize the pathogen if it enters the body in the future.
# The use of vaccines has led to a significant reduction in the incidence and severity of many diseases, including measles, mumps, rubella, and polio. 
```

It can also be used with text-generation-inference

```sh
model=Writer/palmyra-med-20b
volume=$PWD/data

docker run --gpus all --shm-size 1g -p 8080:80 -v $volume:/data ghcr.io/huggingface/text-generation-inference --model-id $model
```

##  Dataset
For the fine-tuning of our LLMs, we used a custom-curated medical dataset that combines data from
two publicly available sources: PubMedQA (Jin et al. 2019) and MedQA (Zhang et al. 2018).The
PubMedQA dataset, which originated from the PubMed abstract database, consists of biomedical
articles accompanied by corresponding question-answer pairs. In contrast, the MedQA dataset
features medical questions and answers that are designed to assess the reasoning capabilities of
medical question-answering systems.
We prepared our custom dataset by merging and processing data from the aforementioned sources,
maintaining the dataset mixture ratios detailed in Table 1. These ratios were consistent for finetuning
both Palmyra-20b and Palmyra-40b models. Upon fine-tuning the models with this dataset, we refer
to the resulting models as Palmyra-Med-20b and Palmyra-Med-40b, respectively.


| Dataset | Ratio    | Count |
| -----------|----------- | ----------- |
| PubMedQA | 75%       | 150,000       |
| MedQA | 25%    | 10,178        |


## Evaluation
we present the findings of our experiments, beginning with the evaluation outcomes of
the fine-tuned models and followed by a discussion of the base models’ performance on each of the
evaluation datasets. Additionally, we report the progressive improvement of the Palmyra-Med-40b
model throughout the training process on the PubMedQA dataset.

| Model | PubMedQA    | MedQA |
| -----------|----------- | ----------- |
| Palmyra-20b | 49.8       | 31.2       |
| Palmyra-40b | 64.8 | 43.1| 
| Palmyra-Med-20b|  75.6 | 44.6| 
| Palmyra-Med-40b|  81.1 | 72.4| 



## Limitation
The model may not operate efficiently beyond the confines of the healthcare field.
Since it has not been subjected to practical scenarios, its real-time efficacy and precision remain undetermined.
Under no circumstances should it replace the advice of a medical professional, and it must be regarded solely as a tool for research purposes.

## Citation and Related Information


To cite this model:
```
@misc{Palmyra-Med-20B,
  author = {Writer Engineering team},
  title = {{Palmyra-Large Parameter Autoregressive Language Model}},
  howpublished = {\url{https://dev.writer.com}},
  year = 2023,
  month = March 
}
```

## Contact
Hello@writer.com
# [Open LLM Leaderboard Evaluation Results](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard)
Detailed results can be found [here](https://huggingface.co/datasets/open-llm-leaderboard/details_Writer__palmyra-med-20b)

| Metric                | Value                     |
|-----------------------|---------------------------|
| Avg.                  | 40.02   |
| ARC (25-shot)         | 46.93          |
| HellaSwag (10-shot)   | 73.51    |
| MMLU (5-shot)         | 44.34         |
| TruthfulQA (0-shot)   | 35.47   |
| Winogrande (5-shot)   | 65.35   |
| GSM8K (5-shot)        | 2.65        |
| DROP (3-shot)         | 11.88         |