test3_sft_16bit / README.md
alnrg2arg's picture
Update README.md
a892e9a verified
metadata
language:
  - en
license: cc-by-nc-4.0
tags:
  - text-generation-inference
  - transformers
  - unsloth
  - mistral
  - trl
base_model: alnrg2arg/blockchainlabs_7B_merged_test2_4
datasets:
  - Open-Orca/SlimOrca

Uploaded model

  • Finetuned from model : alnrg2arg/blockchainlabs_7B_merged_test2_4

This is a SFT version of the model from blockchainlab test 2.4 - alnrg2arg/blockchainlabs_7B_merged_test2_4.

The project is running to make a small LLM for a on-device purpose.

Overall pipeline for this iteration is

1.Merging to make a base model (7B) 2.Prune the model to reduce the parameter (50% sparcity) 3.For recovery phase of the pruning, the DPO is chosen.

This model which is not pruned is intended to compare with the pruned model.

DPO consists of two parts : SFT and DPO - Now this model is the intermediate format (SFT) This model can also be compared to the DPO version of the model.

This is the code and parameters I chose for this model(SFT).

from transformers import TrainingArguments
from trl import SFTTrainer
from datasets import load_dataset
from unsloth import FastLanguageModel, FastMistralModel


max_seq_length = 2048 # Supports automatic RoPE Scaling, so choose any number

# Load model
model, tokenizer = FastMistralModel.from_pretrained(
    model_name = "alnrg2arg/blockchainlabs_7B_merged_test2_4,
    max_seq_length = max_seq_length,
    dtype = None, # None for auto detection. Float16 for Tesla T4, V100, Bfloat16 for Ampere+
    load_in_4bit = True, # Use 4bit quantization to reduce memory usage. Can be False
    #device_map = "balanced"
    # token = "hf_...", # use one if using gated models like meta-llama/Llama-2-7b-hf
)

model = FastMistralModel.get_peft_model(
    model,
    r = 16,
    target_modules = ["q_proj", "k_proj", "v_proj", "o_proj",
                      "gate_proj", "up_proj", "down_proj",],
    lora_alpha = 16,
    lora_dropout = 0, # Dropout = 0 is currently optimized
    bias = "none",    # Bias = "none" is currently optimized
    use_gradient_checkpointing = True,
    random_state = 3407,
    max_seq_length = max_seq_length,
)

The code and parameters are borrowed from https://colab.research.google.com/drive/1SKrKGV-BZoU4kv5q3g0jtE_OhRgPtrrQ?usp=sharing

Benchmark scores

Tasks Version Filter n-shot Metric Value Stderr
arc_challenge 1 none 25 acc 0.7116 ± 0.0132
none 25 acc_norm 0.7346 ± 0.0129
Tasks Version Filter n-shot Metric Value Stderr
hellaswag 1 none 10 acc 0.7222 ± 0.0045
none 10 acc_norm 0.8865 ± 0.0032
Tasks Version Filter n-shot Metric Value Stderr
truthfulqa_mc2 2 none 0 acc 0.7043 ± 0.015
Groups Version Filter n-shot Metric Value Stderr
mmlu N/A none 0 acc 0.6367 ± 0.1258
- humanities N/A none 5 acc 0.5968 ± 0.1122
- other N/A none 5 acc 0.7049 ± 0.1123
- social_sciences N/A none 5 acc 0.7374 ± 0.0774
- stem N/A none 5 acc 0.5309 ± 0.1373
Tasks Version Filter n-shot Metric Value Stderr
winogrande 1 none 5 acc 0.8477 ± 0.0101
Tasks Version Filter n-shot Metric Value Stderr
gsm8k 2 get-answer 5 exact_match 0.7468 ± 0.012

Average 75.94