TinyLlama-1.1B-Chat QLoRA (Alpaca)
LoRA adapter for TinyLlama/TinyLlama-1.1B-Chat-v1.0, trained with QLoRA (4-bit NF4 base,
bf16 adapters) on a 2,000-row slice of Alpaca. A learning exercise, not a pro
Training
| rank / alpha | 16 / 32 |
| target modules | q, k, v, o, gate, up, down |
| LR / schedule | 2e-4, cosine, 3% warmup |
| effective batch | 16 |
| epochs | 1 |
| trainable params | 12.6M (1.15%) |
Loss was computed on assistant turns only, with EOS kept in the loss.
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
base = AutoModelForCausalLM.from_pretrained("TinyLlama/TinyLlama-1.1B-Chat-v1.0")
model = PeftModel.from_pretrained(base, "dasuntm/tinyllama-alpaca-qlora-v1")
tok = AutoTokenizer.from_pretrained("dasuntm/tinyllama-alpaca-qlora-v1")
License
The base model is Apache 2.0, but Alpaca is CC BY-NC 4.0 (non-commercial) — it was
generated with OpenAI's API. This adapter inherits that restriction. Non-comm
- Downloads last month
- 9
Model tree for dasuntm/tinyllama-alpaca-qlora-v1
Base model
TinyLlama/TinyLlama-1.1B-Chat-v1.0