|
--- |
|
license: apache-2.0 |
|
datasets: |
|
- Laurent1/MedQuad-MedicalQnADataset_128tokens_max |
|
library_name: adapter-transformers |
|
tags: |
|
- medical |
|
--- |
|
# Model Card for mpt-7b-instruct2-QLoRa-medical-QA |
|
|
|
 |
|
|
|
<font color="FF0000" size="5"> <b> |
|
This is a QA model for answering medical questions<br /> </b></font> |
|
<br><b>Foundation Model : https://huggingface.co/ibm/mpt-7b-instruct2 <br /> |
|
Dataset : https://huggingface.co/datasets/Laurent1/MedQuad-MedicalQnADataset_128tokens_max <br /></b> |
|
The model has been fine tuned with 2 x GPU T4 (RAM : 2 x 14.8GB) + CPU (RAM : 29GB). <br /> |
|
|
|
|
|
## <b>Model Details</b> |
|
|
|
The model is based upon the foundation model : ibm/mpt-7b-instruct2 (Apache 2.0 License).<br /> |
|
It has been tuned with Supervised Fine-tuning Trainer and PEFT LoRa.<br /> |
|
|
|
### Librairies |
|
<ul> |
|
<li>bitsandbytes</li> |
|
<li>einops</li> |
|
<li>peft</li> |
|
<li>trl</li> |
|
<li>datasets</li> |
|
<li>transformers</li> |
|
<li>torch</li> |
|
</ul> |
|
|
|
|
|
### Notebook used for the training |
|
|
|
https://colab.research.google.com/drive/14nxSP5UuJcnIJtEERyk5nehBL3W03FR3?hl=fr |
|
|
|
=> Improvements can be achieved by increasing the number of steps and using the full dataset. <br /> |
|
|
|
### Direct Use |
|
|
|
|
|
 |
|
|
|
|
|
## <b>Bias, Risks, and Limitations</b> |
|
|
|
In order to reduce training duration, the model has been trained only with the first 5100 rows of the dataset.<br /> |
|
|
|
<font color="FF0000"> |
|
Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model.<br /> |
|
Generation of plausible yet incorrect factual information, termed hallucination, is an unsolved issue in large language models.<br /> |
|
</font> |
|
|
|
## <b>Training Details</b> |
|
|
|
<ul> |
|
<li>per_device_train_batch_size = 1</li> |
|
<li>gradient_accumulation_steps = 16</li> |
|
<li>epoch = 5</li> |
|
<li>2 x GPU T4 (RAM : 14.8GB) + CPU (RAM : 29GB)</li> |
|
</ul> |
|
|
|
### Training Data |
|
|
|
https://huggingface.co/datasets/Laurent1/MedQuad-MedicalQnADataset_128tokens_max |
|
|
|
#### Training Hyperparameters |
|
|
|
|
|
 |
|
|
|
#### Times |
|
|
|
Training duration : 6287.4s |
|
|
|
|
|
 |
