bachir6c/mistral-mam-lora

Mistral-7B Math LoRA Adapter (MAM Project)

This repository contains the LoRA adapter weights for Mistral-7B-v0.1, fine-tuned on the GSM8K mathematical reasoning dataset.

• Developer: Mouhamed Bachir CISSE (Polytech Lyon, Applied Mathematics & Modeling)
• Base Model: mistralai/Mistral-7B-v0.1
• Task: Causal Language Modeling / Mathematical Reasoning
• Fine-tuning Method: Parameter-Efficient Fine-Tuning (PEFT) with LoRA

---

Training Details

• Dataset: openai/gsm8k
• Trained Parameters: 6,815,744 (0.094% of total parameters)
• Training Loss: Decreased from 1.025 to 0.722 (-30%) over 200 steps
• Hardware: 1x NVIDIA T4 GPU
• Precision: BF16 / 4-bit quantization

LoRA Configuration

LoraConfig(
    r=8,
    lora_alpha=16,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj"],
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM"
)

How to Use

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base_model_id = "mistralai/Mistral-7B-v0.1"
adapter_id = "bachir6c/mistral-mam-lora"

tokenizer = AutoTokenizer.from_pretrained(base_model_id)
base_model = AutoModelForCausalLM.from_pretrained(
    base_model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto"
)

# Load the LoRA adapter
model = PeftModel.from_pretrained(base_model, adapter_id)

prompt = "Question: Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May?\nAnswer:"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

outputs = model.generate(**inputs, max_new_tokens=128)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))