PathaShiva/qwen2.5-1.5b-support-qlora

Qwen2.5-1.5B-Instruct — Customer Support (QLoRA adapter)

A LoRA adapter that fine-tunes Qwen/Qwen2.5-1.5B-Instruct to answer customer-support questions in a helpful, on-brand support tone, with correct use of fillable placeholder slots (e.g. {{Order Number}}).

• Base model: Qwen/Qwen2.5-1.5B-Instruct (Apache-2.0)
• Method: QLoRA (4-bit) — r=16, α=32, dropout=0.05, all linear layers
• Data: Bitext customer-support (27 intents), deduplicated + leakage-safe split
• Code / full write-up: https://github.com/PathaShiva/customer-support-slm

Results (held-out, leakage-free test set)

Pairwise LLM-as-judge (llama3.1:8b, order-swapped), tuned vs base: Win 32% · Tie 64% · Loss 4%.

metric	base	tuned
placeholder adherence	6%	89%
fabricated value rate	2%	0%
ROUGE-L vs gold	0.167	0.321

Prompt template (exact)

ChatML (Qwen2.5 native), applied via tokenizer.apply_chat_template. System prompt used in training and inference:

<|im_start|>system
You are a helpful customer support assistant.<|im_end|>
<|im_start|>user
{customer message}<|im_end|>
<|im_start|>assistant

Keep the system prompt identical to the above — the adapter was trained conditioned on it, so changing it degrades quality.

How to load and run

import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base = "Qwen/Qwen2.5-1.5B-Instruct"
tok = AutoTokenizer.from_pretrained(base)
model = PeftModel.from_pretrained(
    AutoModelForCausalLM.from_pretrained(base, torch_dtype=torch.float16),
    "PathaShiva/qwen2.5-1.5b-support-qlora").eval()

msgs = [{"role": "system", "content": "You are a helpful customer support assistant."},
        {"role": "user", "content": "how do I cancel my order?"}]
prompt = tok.apply_chat_template(msgs, tokenize=False, add_generation_prompt=True)
out = model.generate(**tok(prompt, return_tensors="pt"),
                     max_new_tokens=256, do_sample=False)
print(tok.decode(out[0], skip_special_tokens=True))

Recommended decoding: greedy (do_sample=False), max_new_tokens=256.

Training

QLoRA on a free Kaggle T4, 2 epochs, lr 2e-4 (cosine), effective batch 32, max_len 1024, bf16. eval_loss fell 1.09 → 0.71 with a small train/eval gap (no overfitting).

Limitations

• Emits {{placeholder}} slots (e.g. {{Order Number}}) that a backend is expected to fill — it does not look up real values.
• Weakest intent is edit_account; some delivery queries can use slots vaguely.
• English only; trained on synthetic support data (Bitext).
• 1.5B model — for higher quality, the same pipeline supports a larger base.