Mwanzau/Mazgu_Llama-1B-V2-Knowledge

Mazgu_Llama-1B-V2 (LoRA Adapters)

This repository contains fine-tuned LoRA (Low-Rank Adaptation) adapter weights for unsloth/Llama-3.2-1B-Instruct. The model has been trained via Knowledge-SFT to improve instruction adherence, factual grounding, and technical domain fluency in Tumbuka.

🛠️ Usage with PEFT & Unsloth

To load these adapters on top of the base model using unsloth or standard Hugging Face Transformers:

import torch
from unsloth import FastLanguageModel

# 1. Load Base Model & Attach Adapters
model, tokenizer = FastLanguageModel.from_pretrained(
    model_name = "Mwanzau/Mazgu_Llama-1B-V2", # Automatically fetches base + adapters
    max_seq_length = 2048,
    load_in_4bit = True,
)

FastLanguageModel.for_inference(model)

# 2. Formulate Prompt (Alpaca / Instruction Format)
prompt = """### Instruction:
Longosolani vyakurya ivyo vili bwino ku munda.

### Response:
"""

inputs = tokenizer([prompt], return_tensors="pt").to("cuda")

# 3. Generate
outputs = model.generate(
    **inputs,
    max_new_tokens = 150,
    temperature = 0.6,
    repetition_penalty = 1.2
)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

### 🏋️ Training Parameters
Framework: Unsloth / PEFT
Rank ($r$): 16
Alpha ($\alpha$): 16
Training Steps: 500 Knowledge-SFT steps
Dataset Format: Alpaca Instruction-Response (### Instruction: / ### Response:)
Focus: Grounded domain factual recall (Agriculture, literature, Tumbuka linguistics).