Mistral-7B-DSM5

This model is fine-tuned from the Mistral-7b-Instruct-v0.2. We propose a method where an Instruct-Tuned language model first acquires knowledge in a specific domain through RAG (Retrieval-Augmented Generation) techniques. Then, using prompts, the model generates a fine-tuning dataset specific to that domain, which is subsequently used to fine-tune the model itself, transforming it into an expert in that domain.

This pipeline addresses two main challenges:

1. Dynamic Content: For example, in scenarios involving company policies, standards, and regulations, whenever there are content updates, this method can quickly generate new datasets for fine-tuning.
2. Data Scarcity: For some niche or confidential domains, where there is limited data or privacy concerns preventing data from being shared, this method can self-generate datasets for Instruct Tuning, ensuring security and privacy without the fear of data scarcity.

This model, as an example, is focused on psychiatry. We used RAG with documents from the "Desk Reference to the Diagnostic Criteria From DSM-5" to fine-tune Mistral-7B into a psychiatry Q&A model with knowledge of the DSM-5 diagnostic manual.

More Pipeline Detail Please Refer To Citation's Paper

Training Config

Training spend 2 hours in duration on single RTX A6000, by using LoRA.

HyperParameter	Value
Batch Size	2
Learning Rate	5e-5
Epochs	10
Gradient Accumulation	8
LoRA r	16

Disclaimer

This model is intended for research purposes only. The author does not guarantee its accuracy, completeness, or suitability for any purpose. Any commercial or other use requires consultation with a legal professional, and the author assumes no responsibility for such use. Users bear all risks associated with the results of using this model. The author is not liable for any direct or indirect losses or damages, including but not limited to loss of profits, business interruption, or data loss. Any use of this model is considered acceptance of the terms of this disclaimer.

Model Usage

Download model

Here is the example for you to download with huggingface transformers:

from transformers import AutoTokenizer,AutoModelForCausalLM
import torch

tokenizer = AutoTokenizer.from_pretrained("chillymiao/Mistral-7B-DSM5")
model = AutoModelForCausalLM.from_pretrained("chillymiao/Mistral-7B-DSM5")

Citation

@INPROCEEDINGS{10633534,
  author={Sung, Chih-Wei and Lee, Yu-Kai and Tsai, Yin-Te},
  booktitle={2024 IEEE 48th Annual Computers, Software, and Applications Conference (COMPSAC)}, 
  title={A New Pipeline for Generating Instruction Dataset via RAG and Self Fine-Tuning}, 
  year={2024},
  volume={},
  number={},
  pages={2308-2312},
  keywords={Ethics;Technological innovation;Accuracy;Computational modeling;Pipelines;Organizations;Psychiatry;Large Language Model;Retrieval-Augmented Generation;Psychiatry;Instruction Tuning},
  doi={10.1109/COMPSAC61105.2024.00371}}