Shears Model Card: shears-mpt-7b-50-gsm8k-heuristic-adapter

The heuristic adapter discovered from the super-adapter fine-tuned on sparsified MPT-7B with GSM8K datasets using Shears.

Model Details

Information

• Model name: shears-mpt-7b-50-gsm8k-heuristic-adapter
• Base model: IntelLabs/shears-mpt-7b-50-base
• Sparsity: 50%
• Subnetwork version: Heuristic
• NNCF Configuration: nncf_shears_mpt.json

Adapter Configuration

• LoRA rank: 32 (24 in the heuristic subnetwork)
• LoRA alpha: 64
• LoRA target modules: q_proj, k_proj, v_proj, out_proj, up_proj, down_proj
• LoRA rank search space: [32, 24, 16] (for each LoRA module)

Training Hyperparameters

• Batch size: 16
• Learning rate: 3e-4
• Epoch: 5

Training and Evaluation

GSM8K dataset: https://huggingface.co/datasets/gsm8k

How to use

Use our modified PEFT library (apply patch):

git clone https://github.com/huggingface/peft.git
cd peft && git checkout v0.5.0 && git apply --ignore-space-change --ignore-whitespace peft-modifications-for-shears-inference-usage.patch && pip install -e . && cd ..

import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM
from transformers import AutoTokenizer

def generate_prompt(instruction):
    return f"""Below is an instruction that describes a task. Write a response that appropriately completes the request. 

                    ### Instruction:
                    {instruction}

                    ### Response:
                    """

base_model = AutoModelForCausalLM.from_pretrained("IntelLabs/shears-mpt-7b-50-base", trust_remote_code=True)
model = PeftModel.from_pretrained(base_model, "IntelLabs/shears-mpt-7b-50-gsm8k-heuristic-adapter")
model.eval()

non_zero_params = sum([(param.data != 0).sum().item() for _, param in model.named_parameters()])
print(f"Number of all non-zero parameters: {non_zero_params}")

tokenizer = AutoTokenizer.from_pretrained("IntelLabs/shears-mpt-7b-50-base", trust_remote_code=True)

instruction = "Edgar eats 18 pretzels a day. If his brother eats 1/2 as many, how many does his brother eat in a week?"
prompt = generate_prompt(instruction)
inputs = tokenizer(prompt, return_tensors="pt")
input_ids = inputs["input_ids"].to(model.device)
with torch.no_grad():
    generation_output = model.generate(
        input_ids=input_ids,
        return_dict_in_generate=True,
        output_scores=True,
        max_new_tokens=256,
        use_cache=True,
        num_beams=4,
    )
    s = generation_output.sequences[0]
    output = tokenizer.decode(s)
print(output)

Evaluation Results

Model	Sparsity	GSM8K Accuracy
MPT-7B-Shears	50%	33.4

Model Sources

• Repository: https://github.com/IntelLabs/Hardware-Aware-Automated-Machine-Learning/tree/main/Shears
• Paper: Shears: Unstructured Sparsity with Neural Low-rank Adapter Search

Citation

@article{munoz2024shears,
  title = {Shears: Unstructured Sparsity with Neural Low-rank Adapter Search},
  author={J. Pablo Munoz and Jinjie Yuan and Nilesh Jain},
  journal={The 2024 Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL-2024)},
  year={2024}
}

License

Apache-2.0