See axolotl config

axolotl version: 0.4.1

base_model: mistralai/Mistral-7B-v0.1
model_type: MistralForCausalLM
tokenizer_type: LlamaTokenizer
is_mistral_derived_model: true

load_in_8bit: false
load_in_4bit: true
strict: false

lora_fan_in_fan_out: false
data_seed: 49
seed: 49

datasets:
  - path: NimaZahedinameghi/source_injury
    type: alpaca

dataset_prepared_path: last_run_prepared
val_set_size: 0.1
output_dir: ./qlora-alpaca-out
hub_model_id: NimaZahedinameghi/source_of_injury

adapter: qlora
lora_model_dir:

sequence_len: 896
sample_packing: false
pad_to_sequence_len: true

lora_r: 32
lora_alpha: 16
lora_dropout: 0.05
lora_target_linear: true
lora_fan_in_fan_out:
lora_target_modules:
  - gate_proj
  - down_proj
  - up_proj
  - q_proj
  - v_proj
  - k_proj
  - o_proj

wandb_project: source_of_injury
wandb_entity: uqam

gradient_accumulation_steps: 4
micro_batch_size: 16
eval_batch_size: 16
num_epochs: 3
optimizer: adamw_bnb_8bit
lr_scheduler: cosine
learning_rate: 0.0002
max_grad_norm: 1.0
adam_beta2: 0.95
adam_epsilon: 0.00001
save_total_limit: 12

train_on_inputs: false
group_by_length: false
bf16: true
fp16: false
tf32: false

gradient_checkpointing: true
early_stopping_patience:
resume_from_checkpoint:
local_rank:
logging_steps: 1
xformers_attention:
flash_attention: true

loss_watchdog_threshold: 5.0
loss_watchdog_patience: 3

warmup_steps: 20
evals_per_epoch: 4
eval_table_size:
eval_table_max_new_tokens: 128
saves_per_epoch: 6
debug:
weight_decay: 0.0
fsdp:
fsdp_config:
special_tokens:
  bos_token: "<s>"
  eos_token: "</s>"
  unk_token: "<unk>"
save_safetensors: true

source_of_injury

This model is built by Nima Zahedinameghi. It's a fine-tuned version of mistralai/Mistral-7B-v0.1 on the source_injury dataset. It achieves the following results on the evaluation set:

• Loss: 0.5867, after 3 epochs.

Important: The model is still underdeveloped. It has been trained on a tiny synthetic data with limited medical labels. It's capabilities are limited. I will release the new version of the model soon.

How to use the model

1. Pip install the required dependencies

transformers==4.36.2
datasets==2.15.0
peft==0.6.0
accelerate==0.24.1
bitsandbytes==0.41.3.post2
safetensors==0.4.1
scipy==1.11.4
sentencepiece==0.1.99
protobuf==4.23.4 --upgrade

2. Load the Model and Tokenizer:

   from peft import AutoPeftModelForCausalLM
   from transformers import AutoTokenizer
   
   model_id = 'NimaZahedinameghi/source_of_injury'
   model = AutoPeftModelForCausalLM.from_pretrained(model_id).cuda()
   tokenizer = AutoTokenizer.from_pretrained(model_id)
   tokenizer.pad_token = tokenizer.eos_token
   

3. Define the Prompt Function: Create a function to structure your prompt correctly:

   def prompt(incident_description):
       return f"""[INST] <<SYS>>
       The claims are managed by reviewing incident descriptions submitted by workers. Claim coders review the incident description and populate a database with reasoning towards determining the source of injury (InjurySource).
       <</SYS>>
   
       IncidentDescription: {incident_description}
       [/INST]
       """
   
   def prompt_tok(incident_description):
       _p = prompt(incident_description)
       input_ids = tokenizer(_p, return_tensors="pt", truncation=True).input_ids.cuda()
       out_ids = model.generate(input_ids=input_ids, max_new_tokens=500, do_sample=False)
       return tokenizer.batch_decode(out_ids.detach().cpu().numpy(), skip_special_tokens=True)[0][len(_p):]
   

4. Make Predictions: Use the function to get predictions from your model:

   incident_description = "While working on a vehicle repair, I had to contort my body to access hard-to-reach areas. This position caused severe discomfort and pain in my neck and shoulders."
   output = prompt_tok(incident_description)
   print(output)
   

This function will take an incident description and return the reasoning and injury source as determined by your fine-tuned model. Ensure you follow the specific prompt format that matches your training setup.

Model description

the model is fine tuned on a small synthetic dataset with 4bit precision. The model is still underdeveloped and covers very limited number of medical labels.

Intended uses & limitations

Further testing is required to evaluate the model performance on custome evaluations

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0002
• train_batch_size: 16
• eval_batch_size: 16
• seed: 49
• gradient_accumulation_steps: 4
• total_train_batch_size: 64
• optimizer: Adam with betas=(0.9,0.95) and epsilon=1e-05
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 20
• num_epochs: 3

Training results

Training Loss	Epoch	Step	Validation Loss
2.0189	0.1481	1	2.0511
2.0285	0.2963	2	2.0442
1.9559	0.5926	4	1.9259
1.802	0.8889	6	1.6212
1.4115	1.1852	8	1.2261
1.1171	1.4815	10	1.0004
0.9691	1.7778	12	0.8657
0.747	2.0741	14	0.7082
0.6407	2.3704	16	0.6205
0.6101	2.6667	18	0.5867

Framework versions

• PEFT 0.11.1
• Transformers 4.42.3
• Pytorch 2.1.2+cu118
• Datasets 2.19.1
• Tokenizers 0.19.1