Qwen2.5-0.5B Structured-Answer LoRA
Live demo: https://huggingface.co/spaces/Meow07/format-lora-demo Code and dataset: https://github.com/Aryan-1947/qwen-format-lora
A LoRA adapter that makes Qwen2.5-0.5B-Instruct answer in a fixed format: Summary / Key points (3 bullets) / Next step.
Training
- Method: LoRA (r=16, alpha=32) on all attention + MLP projections, fp32 base, free Colab T4
- Data: 440 train / 48 val examples, self-distilled. Questions from HuggingFaceH4/no_robots; answers generated by Qwen2.5-1.5B-Instruct under a format-enforcing system prompt, filtered and normalized, then trained without the system prompt.
- 2 epochs, lr 2e-4, effective batch 16. Final val loss 1.17.
Results
Format compliance on 40 unseen questions: 39/40 (fine-tuned) vs 0/40 (base).
Limitations
- Teaches format, not facts: answers can be wrong (the 0.5B base and 1.5B teacher make mistakes).
- The training data was generated by a 1.5B model and contains factual errors, so it should not be used as a knowledge source.
- Occasional stray non-English tokens.
- Small dataset, single run, no human evaluation.
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-0.5B-Instruct")
model = PeftModel.from_pretrained(base, "Meow07/qwen2.5-0.5b-format-lora")
- Downloads last month
- 59
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support