Nanthasit/sakthai-plus-1.5b-lora

SakThai Plus 1.5B — LoRA Adapter

Unmerged rsLoRA adapter — the training-time weights behind SakThai Plus 1.5B
Qwen2.5-1.5B-Instruct · rsLoRA r=16 · all 7 linear modules · 74 MB

---

Model Description

This repository contains the unmerged LoRA adapter for SakThai Plus 1.5B — the next-generation tool-calling model of the SakThai family. It is a rank-stabilized LoRA (rsLoRA) adapter trained with SFT via TRL on the SakThai tool-calling dataset family (v7 + v8, 2,965 rows, published as sakthai-combined-v10). Row count line-counted from data/train.jsonl on 2026-07-31.

The adapter adds tool-calling / function-calling behaviour on top of Qwen/Qwen2.5-1.5B-Instruct. The merged, drop-in full-weight checkpoint lives at Nanthasit/sakthai-plus-1.5b; this repo keeps the raw adapter for anyone who wants to inspect it, re-train from it, or load it with PEFT.

Use this repo if you want:

• The exact training-time LoRA weights (74 MB instead of a 3.09 GB merged checkpoint)
• To re-train / continue fine-tuning from the adapter
• To merge it yourself into the base model (snippet below)

Use the merged repo instead if you want: plain AutoModelForCausalLM loading without a PEFT dependency.

---

⚠️ Important: Inference Notes

This is a PEFT LoRA adapter, not a standalone model:

• ❌ Not supported by the HF serverless Inference API — verified 2026-07-31: providers return 400 Model not supported / 403 Not enabled for adapter repos. You must run it locally, or deploy via TGI with --lora-adapters.
• ✅ Local inference works: load the base model, then attach this adapter with PeftModel.from_pretrained (see below).
• ✅ Merge it into the base to get a standalone checkpoint (see merge snippet).

---

Quick Start (PEFT)

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base_model_id = "Qwen/Qwen2.5-1.5B-Instruct"
adapter_id = "Nanthasit/sakthai-plus-1.5b-lora"

# 1. Load the base model
base = AutoModelForCausalLM.from_pretrained(
    base_model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)

# 2. Attach the adapter
model = PeftModel.from_pretrained(base, adapter_id)
tokenizer = AutoTokenizer.from_pretrained(adapter_id)

# 3. Chat
messages = [
    {"role": "system", "content": "You are SakThai-Agent, a helpful assistant. When needed, call tools using <tool_call> blocks."},
    {"role": "user", "content": "What's the weather in Bangkok?"},
]
inputs = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=256, temperature=0.3)
print(tokenizer.decode(outputs[0][inputs.shape[1]:], skip_special_tokens=True))

Tool-Calling Format

The chat template includes function-calling XML tool support. Provide tools as a <tools> block in the system prompt and let the model emit <tool_call> JSON:

<tools>
{"name": "get_weather", "description": "Get current weather", "parameters": {"type": "object", "properties": {"location": {"type": "string"}}, "required": ["location"]}}
</tools>

Expected model output when the user asks about weather:

<tool_call>
{"name": "get_weather", "arguments": {"location": "Bangkok"}}
</tool_call>

Note: The <tools> block is required for reliable function calling — the model was trained on this exact XML format. Omitting it makes the model fall back to answering directly.

---

Merge to Full Weights

merged = model.merge_and_unload()
merged.save_pretrained("./sakthai-plus-1.5b-merged")
tokenizer.save_pretrained("./sakthai-plus-1.5b-merged")

After merging, you can load the result with plain AutoModelForCausalLM and convert it to GGUF for llama.cpp / Ollama. (The official merged build is already published at sakthai-plus-1.5b — merging locally is only needed for re-training or custom quantization.)

---

Training Details

Detail	Value
Base model	Qwen/Qwen2.5-1.5B-Instruct
Method	rsLoRA (rank-stabilized LoRA) + SFT (TRL)
LoRA rank (r)	16
LoRA alpha	32
LoRA dropout	0.05
Target modules	All 7 linear projections: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
Training data	sakthai-combined-v7 + sakthai-combined-v10 (2,965 rows)
Context window	32,768 tokens
Precision	BF16

Training Hyperparameters (verified in training_args.bin)

Setting	Value
Epochs	3
Learning rate	2e-4 (AdamW)
Per-device batch size	2
Gradient accumulation	8
Optimizer	adamw_8bit
LR scheduler	cosine
Warmup	0.03
Weight decay	0.0
Max grad norm	1.0
Seed	42
Precision	BF16 (bf16: true, fp16: false)
Gradient checkpointing	on
Output dir	./sakthai-1.5b-lora

All values byte-scanned from the repo's own training_args.bin (SFTConfig pickle) on 2026-07-31 — hardware (T4) is from the reported run, not stored in the file.

Adapter Configuration (adapter_config.json)

Setting	Value
PEFT type	LORA
PEFT version	0.20.0
Rank (r)	16
Alpha	32
Dropout	0.05
rsLoRA	true (use_rslora)
Target modules	All 7 linear projections
Bias	none
Task type	CAUSAL_LM
Trainable params	~37M (36,955,556)
Adapter size	73,911,112 B (73.9 MB, BF16)

---

Architecture

Base architecture verified from Qwen/Qwen2.5-1.5B-Instruct config.json (fetched live 2026-07-31; adapter repos carry no config.json of their own):

Property	Value
Architecture	Qwen2ForCausalLM
Hidden size	1,536
Layers	28
Attention heads	12
KV heads (GQA)	2
Intermediate size	8,960
Vocab size	151,936
Context window	32,768 (RoPE θ 1e6)
Total params	1.54B (1,543,714,304)
Adapter dtype	BF16

---

Benchmarks

✅ Verified 3/3 tool-calling (2026-07-31) — measured on the merged checkpoint (sakthai-plus-1.5b), which is exactly this adapter's weights merged onto the base. Source: .eval_results/benchmark-20260731_043634.yaml on the merged repo:

Metric	Result
Tool call present	3/3 (100%)
Valid JSON	3/3 (100%)
Correct answer	3/3 (100%)
Backend	llama.cpp (GGUF Q4_K_M, local CPU)
Scenario	send_email (SakThai Bench v2 prompt)

Scope note: this is a single-scenario sanity benchmark (3 seeds: 7, 42, 1337), not the full 500-row bench-v2 suite. Full-suite results remain pending and will be published when the harness completes.

Additional repo-verified status (this repo's own .eval_results/, 2026-07-31):

• ✅ Adapter loads correctly with PeftModel (local, verified 2026-07-31)
• ✅ Chat template present with function-calling XML tool support
• ✅ 74 MB adapter size — typical for LoRA on a 1.5B base
• ⚠️ Serverless Inference API does not support LoRA adapter repos (documented above)

Recommended full evaluation framework: sakthai-bench-v2 — 500 rows covering simple, parallel, irrelevance, and multi-turn tool-calling scenarios with held-out tools.

---

Ecosystem Status

From the repo's own .eval_results/cron-eval-sakthai-plus-1.5b-lora-2026-07-31-1.yaml (2026-07-31 06:18Z, cron run 18):

Metric	Value
Health score	44.6/100 (popularity 0 · momentum 0 · benchmarks 50 · card quality 93 · hygiene 90)
Downloads rank	12/19 (8 siblings tied at 0 — day-1 repo)
Download velocity	0.0 dl/day
Card quality	93/100 (model-index present; -7 for verified-flag mismatch: API cardData reports verified: false while raw README claims verified: true with no verifyToken — known Hub normalization, same pattern as sibling cards)
Repo hygiene	90/100 (12 eval YAML files, inconsistent timestamp naming — minor)

Earlier health-check (health-check-sakthai-plus-1.5b-lora-2026-07-31.yaml, slim scorer): 51/100 — author rank 16/19, momentum 50 (velocity rank 1 among new repos), card quality 75, hygiene 80. Both scorers agree the path to lift health: promote the merged sibling, publish a GGUF of the merged checkpoint, and add a verified full-suite bench-v2 eval with a real verifyToken.

---

SakThai Model Family

One of 20 public model repos in the SakThai Model Family collection (plus a private experimental embedding, plus companion repos sakthai-bench-v3 and sakthai-pipeline). Live download counts as of 2026-07-31; sizes API-verified from largest weight file.

Model	Size	Type	Downloads
Plus 1.5B LoRA (this model) ⬅	73.9 MB	LoRA adapter (you are here)	0
Context 1.5B Merged	3.09 GB	Merged weights (flagship)	1,599
Context 0.5B Merged	988 MB	Merged weights (edge)	1,370
Context 7B Merged	15.2 GB	Merged weights (power)	744
Context 7B 128K	config-only	YaRN long-context recipe	506
Context 7B Tools	20.2 MB	LoRA adapter	399
Embedding Multilingual	471 MB	Embedding model	362
Context 1.5B Tools v1	~8.75 MB	LoRA adapter (v1)	349
Vision 7B	4.08 GB	Vision-language (GGUF)	186
TTS Model	141 MB	Kokoro-82M TTS (GGUF)	150
Context 0.5B Tools	988 MB	Merged weights	94
Coder 1.5B	1.12 GB	Code model (GGUF)	93
Context 1.5B Merged v2	3.09 GB	Merged weights	0
Context 1.5B Tools v2	73.9 MB	LoRA adapter	0
Coder Browser GGUF	7.11 GB	Browser agent F16 GGUF	0
Coder Browser LoRA	73.9 MB	LoRA adapter	0
Coder Browser	3.09 GB	Browser agent (merged)	0
Plus 1.5B	3.09 GB	Merged weights	0
Plus 1.5B Coder	—	Planned (no weights yet)	0

Sizes API-verified from largest weight file (2026-07-31). "0" = released recently, counts still low.

---

Limitations

• LoRA adapters cannot be served via the HF serverless Inference API — deploy locally or merge first.
• Training focused on tool-calling fidelity; general open-ended chat may be weaker than the instruct base.
• Best results require the documented <tools> XML prompt format.
• 1.5B parameter scale limits complex multi-step reasoning vs. larger models.
• Benchmark coverage is a single verified scenario (3/3) — the full bench-v2 suite is still pending.
• Adapter's model-index score is inherited from the merged checkpoint — the adapter itself cannot be benchmarked standalone.

---

Citation

If you use this model or the SakThai family in your work, please cite:

@misc{sakthai-plus-1.5b-lora,
  author = {Beer (beer-sakthai) and the SakThai Agent family},
  title = {SakThai Plus 1.5B LoRA Adapter -- rsLoRA Tool-Calling Fine-Tune},
  year = {2026},
  publisher = {Hugging Face},
  howpublished = {\url{https://huggingface.co/Nanthasit/sakthai-plus-1.5b-lora}},
  note = {Fine-tune of Qwen2.5-1.5B-Instruct for structured tool-calling}
}

@article{qwen25,
  title={Qwen2.5 Technical Report},
  author={Qwen Team and An Yang and Baosong Yang and Beichen Zhang and et al.},
  journal={arXiv preprint arXiv:2412.15115},
  year={2024}
}

@article{rslora,
  title={RsLoRA: A Rank-Stabilized LoRA that Outperforms Standard LoRA},
  author={Damjan Kalajdzievski},
  journal={arXiv preprint arXiv:2312.03732},
  year={2023}
}

---

Related

• Merged checkpoint: SakThai Plus 1.5B
• Collection: SakThai Model Family
• Datasets: combined-v7 · combined-v10 · sakthai-bench-v2
• Author: Beer (beer-sakthai) — SakThai Agent family