Model Card for mpt-7b-instruct2-QLoRa-medical-QA
This is a QA model for answering medical questions
Foundation Model : https://huggingface.co/ibm/mpt-7b-instruct2
Dataset : https://huggingface.co/datasets/Laurent1/MedQuad-MedicalQnADataset_128tokens_max
The model has been fine tuned with 2 x GPU T4 (RAM : 2 x 14.8GB) + CPU (RAM : 29GB).
Model Details
The model is based upon the foundation model : ibm/mpt-7b-instruct2 (Apache 2.0 License).
It has been tuned with Supervised Fine-tuning Trainer and PEFT LoRa.
Librairies
- bitsandbytes
- einops
- peft
- trl
- datasets
- transformers
- torch
Notebook used for the training
You can find it in the files and versions tab or : https://colab.research.google.com/drive/14nxSP5UuJcnIJtEERyk5nehBL3W03FR3?hl=fr
=> Improvements can be achieved by increasing the number of steps and using the full dataset.
Direct Use
Bias, Risks, and Limitations
In order to reduce training duration, the model has been trained only with the first 5100 rows of the dataset.
Generation of plausible yet incorrect factual information, termed hallucination, is an unsolved issue in large language models.
Training Details
- per_device_train_batch_size = 1
- gradient_accumulation_steps = 16
- epoch = 5
- 2 x GPU T4 (RAM : 14.8GB) + CPU (RAM : 29GB)
Training Data
https://huggingface.co/datasets/Laurent1/MedQuad-MedicalQnADataset_128tokens_max
Training Hyperparameters
Times
Training duration : 6287.4s
- Downloads last month
- 1