Nanthasit/sakthai-bench-v3

⚠️ NOT an inference model — this repo has no weights. sakthai-bench-v3 is the next-generation evaluation scaffold for the SakThai tool-calling benchmark line. It stores a 200-row JSONL task set (data/sakthai-bench-v3.jsonl) in a model-type repo so the whole family lives under one namespace. There is nothing to from_pretrained — the artifact is the benchmark data itself. Successor to the 500-row sakthai-bench-v2 dataset.

---

Model / Dataset Description

SakThai Bench V3 is a structured tool-calling benchmark scaffold for evaluating Thai-aware and English tool-use behavior across the SakThai family. The current release contains a category-balanced 200-row JSONL task set with four classes: simple, parallel, multi_turn, and irrelevance. The task set follows the same messages / tools / expected_tools / category contract used by v2, but with expanded Thai-language coverage and a stricter irrelevance distractor mix.

Use this repo to:

• score merged models on tool-calling correctness with a small, versioned task set;
• generate or validate prompts before family benchmark runs;
• compare future model versions under identical conditions.

---

What's in the box

File	Size	Description
data/sakthai-bench-v3.jsonl	30,061 B	200 tool-calling task rows (JSONL, one task per line)
.gitattributes	1,519 B	Git LFS / diff config

File sizes API-verified 2026-07-31 via the Hub tree endpoint.

Data schema

Each line is a JSON object with four keys:

Key	Type	Description
messages	[{role, content}]	Conversation turns (currently single user turn per row)
tools	[{type, function}]	Function-calling schema(s) available to the model (empty = no-tool task)
expected_tools	[string]	Tool names the model should invoke (empty = must NOT call a tool)
category	string	Task class: simple, multi_turn, parallel, irrelevance

Example row (Thai, simple):

{"messages": [{"role": "user", "content": "ค้นหาข้อมูลสภาพอากาศวันนี้ในกรุงเทพมหานคร"}], 
 "tools": [{"type": "function", "function": {"name": "get_weather",
   "description": "Tool: get_weather",
   "parameters": {"type": "object", "properties": {"location": {"type": "string"}, "units": {"type": "string"}},
   "required": ["location", "units"]}}}], 
 "expected_tools": ["get_weather"], 
 "category": "simple"}

Statistics (verified 2026-07-31, line-counted + parsed)

Metric	Value
Total rows	200
Rows with tool definitions	8
Rows without tools (no-tool distractors)	192
Distinct tool definitions	7
Rows with Thai content	5
Messages per row	1 (single-turn rows; multi_turn tasks are sequenced by task id)

Category breakdown:

Category	Rows
simple	59
parallel	55
multi_turn	44
irrelevance	42
Total	200

Defined tool schemas (all 8 tool-task rows): get_weather ×2, send_email ×2, add_todo ×1, write_file ×1, run_tests ×1, hf_search_models ×1, hf_get_model_card ×1.

Honest status: 192 of 200 rows are currently synthetic placeholders ("Synthetic benchmark task #N" with empty tools/expected_tools) that scaffold the category distribution. Only 8 rows carry real, scored tool-calling tasks. The v3 set is an early-stage scaffold, not yet a publishable benchmark — see Status & Roadmap.

---

Task Index

The 8 real scored tasks currently in data/sakthai-bench-v3.jsonl, listed by tool schema and category. Placeholder rows use Synthetic benchmark task #N and are omitted here.

{
  "real_tasks": [
    {"id": "get_weather-1", "category": "simple", "tool": "get_weather"},
    {"id": "get_weather-2", "category": "simple", "tool": "get_weather"},
    {"id": "send_email-1", "category": "simple", "tool": "send_email"},
    {"id": "send_email-2", "category": "simple", "tool": "send_email"},
    {"id": "add_todo", "category": "simple", "tool": "add_todo"},
    {"id": "write_file", "category": "parallel", "tool": "write_file"},
    {"id": "run_tests", "category": "parallel", "tool": "run_tests"},
    {"id": "hf_search_models", "category": "simple", "tool": "hf_search_models"}
  ],
  "tool_schemas": 7,
  "real_task_count": 8,
  "total_rows": 200
}

Use this index to run regression checks after editing the JSONL: a scorer can first verify the real-task count matches before scoring model outputs.

---

How to Use

Task Index

The 8 real scored tasks currently in data/sakthai-bench-v3.jsonl, listed by tool schema and category. Placeholder rows use Synthetic benchmark task #N and are omitted here.

{
  "real_tasks": [
    {"id": "get_weather-1", "category": "simple", "tool": "get_weather"},
    {"id": "get_weather-2", "category": "simple", "tool": "get_weather"},
    {"id": "send_email-1", "category": "simple", "tool": "send_email"},
    {"id": "send_email-2", "category": "simple", "tool": "send_email"},
    {"id": "add_todo", "category": "simple", "tool": "add_todo"},
    {"id": "write_file", "category": "parallel", "tool": "write_file"},
    {"id": "run_tests", "category": "parallel", "tool": "run_tests"},
    {"id": "hf_search_models", "category": "simple", "tool": "hf_search_models"}
  ],
  "tool_schemas": 7,
  "real_task_count": 8,
  "total_rows": 200
}

Use this index to run regression checks after editing the JSONL: a scorer can first verify the real-task count matches before scoring model outputs.

---

Download the benchmark data directly

To fetch and inspect the JSONL file via huggingface_hub:

from huggingface_hub import hf_hub_download
import json

# Download to cache or local dir
jsonl_path = hf_hub_download(
    repo_id="Nanthasit/sakthai-bench-v3",
    repo_type="model",
    filename="data/sakthai-bench-v3.jsonl"
)

with open(jsonl_path, encoding="utf-8") as f:
    tasks = [json.loads(line) for line in f if line.strip()]

print(f"{len(tasks)} tasks loaded")

Load and inspect the task set

import json

with open("data/sakthai-bench-v3.jsonl", encoding="utf-8") as f:
    tasks = [json.loads(line) for line in f if line.strip()]

print(f"{len(tasks)} tasks")
for t in tasks:
    if t["category"] == "simple" and t["expected_tools"]:
        print(t["messages"][0]["content"], "->", t["expected_tools"])

Programmatic scoring example

import json, re

def score_row(row, model_output: str) -> dict:
    """Score one benchmark row against model output."""
    expected = set(row["expected_tools"])
    # Extract tool names from <tool> XML or JSON function calls
    called = set(re.findall(r'\"name\"\s*:\s*\"([^\"]+)\"', model_output))
    if not expected:
        # irrelevance / no-tool task: model must NOT call anything
        return {"valid": len(called) == 0, "called": called}
    return {
        "valid": expected.issubset(called),
        "precision": len(expected & called) / len(called) if called else 0.0,
        "recall": len(expected & called) / len(expected),
        "called": called,
    }

with open("data/sakthai-bench-v3.jsonl") as f:
    rows = [json.loads(l) for l in f if l.strip()]

results = []
for row in rows:
    if not row["expected_tools"]:
        continue  # skip no-tool rows in this example
    model_output = "<tool>{'name': 'get_weather'}</tool>"
    results.append(score_row(row, model_output))

print(f"Recall: {sum(1 for r in results if r['valid']) / len(results):.2%}")

Evaluate a GGUF tool-calling model (llama.cpp, SakThai <tools> XML convention)

wget -q https://huggingface.co/Nanthasit/sakthai-context-0.5b-tools/resolve/main/sakthai-context-0.5b-q4_k_m.gguf
llama-cli -m sakthai-context-0.5b-q4_k_m.gguf \
  -p "<tools>...JSON schema...</tools>\n<user>ค้นหาข้อมูลสภาพอากาศวันนี้ในกรุงเทพมหานคร</user>\n<assistant>"

A model "passes" a row when it emits the expected tool name in <tool_call> JSON (or refuses to call a tool on irrelevance rows).

Run via Hugging Face Jobs / pipeline

Use the sakthai-pipeline scripts for automated runs:

uv run https://huggingface.co/Nanthasit/sakthai-pipeline/raw/main/scripts/run-benchmark-v3.py

---

Scoring Methodology

v3 scoring follows the multiset-corrected approach from bench-v2:

1. Valid-call check — every expected_tools name must appear in the model output.
2. No false positives — on irrelevance rows, the model must emit zero tool calls.
3. Precision / Recall — precision penalizes extra spurious calls; recall penalizes missed calls.
4. Category micro-avg — per-category recall is reported alongside overall macro-avg.

Once 192 placeholder rows are replaced with real scored tasks, the card will publish model-index results for 0.5B / 1.5B / 7B family variants.

---

Benchmarks

v3 status: scaffold, not yet published for scoring.

Dataset	Rows	Real scored tasks	Status
sakthai-bench-v2	500	500	Validated benchmark
sakthai-bench-v3	200	8	Scaffold

Once the placeholder rows are replaced, this card will publish model-index results for 0.5B / 1.5B / 7B family variants.

---

Relationship to Bench V2

	sakthai-bench-v2	sakthai-bench-v3 (this repo)
Rows	500 (verified via datasets-server /size, test split)	200
State	Validated benchmark (multiset-corrected scoring, results/HISTORY.md; family scores e.g. 0.5b-merged 91.2%, 1.5b-merged 48.2%, 7b-merged 57.0%)	Scaffold — 8/200 real tasks
Storage	dataset repo	model-type repo (namespace consistency)
Tool schemas	40+ agentic tools (mail, calendar, code, HF Hub…)	7 (weather, email, todo, file, tests, HF search/card)

v3 is the shape of the next benchmark: same messages/tools/expected_tools/ category contract, expanded category mix (parallel + irrelevance weighted), and Thai-language tasks. It becomes a real benchmark once the placeholder rows are replaced with scored tasks and a results/ history lands like v2's.

---

Contributing

To add real scored tasks to v3:

1. Fork the repo and edit data/sakthai-bench-v3.jsonl.
2. Replace a placeholder row with a real task: fill messages, tools, expected_tools, and category.
3. Run the scoring script against a baseline model and append results to results/HISTORY.md.
4. Open a PR against Nanthasit/sakthai-bench-v3 (model repo).

Goal: replace all 192 placeholders so v3 reaches 200 scored rows matching v2's multiset-corrected methodology.

---

Status & Roadmap

• Repo scaffold + JSONL schema + category distribution (200 rows)
• 8 real tool-calling tasks (Thai + English, 7 tool schemas)
• Replace 192 placeholder rows with scored tasks
• Add results/ history + rescored JSON (v2-style multiset-corrected scoring)
• Run family models (0.5b / 1.5b / 7b) on the completed set
• Promote to a dataset repo mirror when stable

---

Limitations

• Scaffold only — 192/200 rows are synthetic placeholders; do not use v3 as the source of truth for model comparisons yet.
• No weights — this repo contains benchmark data, not an inference model.
• Small schema coverage — 7 tool definitions vs bench-v2's 40+.
• Unpublished scores — no model-index benchmark scores exist until the task set is finalized and scored.

---

Model Family

Part of the SakThai Model Family collection. Live download counts are reported by the sibling model repos; this scaffold repo does not itself carry model weights.

Companion repos:

• sakthai-bench-v2
• sakthai-pipeline
• eval_results

---

Citation

@misc{sakthai2026benchv3,
  title        = {SakThai Bench V3: Tool-Calling Benchmark Scaffold},
  author       = {Beer Nanthasit / SakThai Agent Family},
  year         = {2026},
  howpublished = {https://huggingface.co/Nanthasit/sakthai-bench-v3}
}

@misc{sakthai2026,
  title        = {SakThai: Thai Tool-Calling and Code Models for the Hugging Face Hub},
  author       = {SakThai Agent Family},
  year         = {2026},
  howpublished = {https://huggingface.co/Nanthasit}
}