Nanthasit/sakthai-pipeline

🏭 SakThai Pipeline

Automation hub of the SakThai Agent family — keeps every model healthy, benchmarked, and documented.

Table of Contents

• What Is This Repo
• Model Description
• How to Use
• Benchmarks & Health
• Scheduled Jobs
• SakThai Model Family
• Limitations
• Citation

What Is This Repo

This repo is not a machine-learning model. It is the automation hub that keeps the SakThai family healthy on the Hugging Face Hub.

It provides a reproducible, zero-cost workflow for:

• probing live inference health across models,
• generating and publishing benchmark data,
• keeping model cards synced with real results,
• scheduling recurring jobs via Hugging Face Jobs.

Everything here runs with free Hugging Face APIs (REST, Inference Router, Jobs) and local compute. No paid GPU. No paid endpoints.

Model Description

This repo stores orchestration code, not model weights.

The SakThai Pipeline exists because family scale outgrew manual card maintenance. As new models ship — LoRA adapters, merged GGUF, coder variants, long-context recipes — health status and benchmark scores drift out of sync. This repo centralizes maintenance so a single command updates health, scores, and markdown across the family.

What it does:

Capability	Script	Output
Router health probing	scripts/health-check.py	health-<date>.yaml in Nanthasit/eval_results
Tool-call evaluation	scripts/run-eval.py	Per-model scored YAML + all summary
Deterministic benchmark generation	scripts/run-benchmark-v3.py	sakthai-bench-v3.jsonl
Card refresh from artifacts	scripts/update-model-cards.py	Updated benchmark tables in READMEs
One-shot automation	scripts/sakthai-do-all.sh	Full cycle in one run

Repo contents (verified 2026-07-31): 10 scripts under scripts/ (~31 KB total), README.md, .gitattributes. Zero weight files.

How to Use

Prerequisites

• Python 3.11+
• uv for dependency resolution
• A Hugging Face token with write access to Nanthasit/* when using --publish

Run Everything

git clone https://huggingface.co/Nanthasit/sakthai-pipeline
cd sakthai-pipeline
bash scripts/sakthai-do-all.sh --publish

Run a Single Step

# Health check only
uv run scripts/health-check.py --publish

# Eval a single model
uv run scripts/run-eval.py --model Nanthasit/sakthai-plus-1.5b --publish

# Regenerate Benchmark v3 deterministically
uv run scripts/run-benchmark-v3.py

# Refresh model cards from latest eval_results
uv run scripts/update-model-cards.py

Run Remotely Without Cloning

uv run https://huggingface.co/Nanthasit/sakthai-pipeline/raw/main/scripts/update-model-cards.py

Quick Validation

# Show current health snapshot
uv run scripts/health-check.py

# List generated reports
ls -lh eval_results/

Benchmarks & Health

Scoring

• Exact tool set match: 1.0
• Partial overlap: 0.5
• Wrong tool selection: 0.0
• Irrelevant request correctly rejected: 1.0

Inference Settings

• Endpoint: api-inference.huggingface.co
• Temperature: 0.1
• Max tokens: 2048

Coverage

Health-checked (14 models):

sakthai-context-0.5b-merged · sakthai-context-0.5b-tools · sakthai-context-1.5b-merged · sakthai-context-1.5b-merged-v2 · sakthai-context-1.5b-tools · sakthai-context-1.5b-tools-v2 · sakthai-context-7b-merged · sakthai-context-7b-tools · sakthai-plus-1.5b · sakthai-coder-1.5b · sakthai-coder-browser · sakthai-vision-7b · sakthai-tts-model · sakthai-embedding-multilingual

Benchmarked (7 merged models):

sakthai-context-0.5b-merged · sakthai-context-1.5b-merged · sakthai-context-1.5b-merged-v2 · sakthai-context-7b-merged · sakthai-plus-1.5b · sakthai-coder-1.5b · sakthai-coder-browser

Excluded by design: sakthai-context-7b-128k is config-only; LoRA adapters are not standalone serverless; sakthai-plus-1.5b-coder is planned without weights yet.

Scheduled Jobs

Use hf jobs run --config scripts/sakthai-jobs.yaml to inspect or trigger scheduled runs.

Job	Schedule	Description
nightly-benchmark	Daily 02:00 UTC	Eval merged models + publish results
health-check	Every 6 hours	Router probe across probed models
model-card-update	Daily 02:30 UTC	Rewrite benchmark tables from latest results
benchmark-v3-publish	Weekly Monday 03:00 UTC	Regenerate and upload sakthai-bench-v3.jsonl
grpo-training	Weekly Sunday 04:00 UTC	CUDA GRPO RL job; staged for GPU environments

Notes:

• The GRPO job references a CUDA image and will not run on free CPU-only HF Jobs. grpo-train.py is staged, not yet committed.
• Health checks exclude LoRA adapters, config-only recipes, and planned repos without weights.

SakThai Model Family

This pipeline serves the whole family — 22 public models in the SakThai Model Family collection (API-verified 2026-08-01).

Model	What it is	Size	Downloads
sakthai-context-1.5b-merged	text-generation	1,894
sakthai-context-0.5b-merged	text-generation	1,730
sakthai-context-7b-merged	text-generation	1,055
sakthai-embedding-multilingual	embedding	651
sakthai-context-7b-128k	text-generation	643
sakthai-context-7b-tools	text-generation	527
sakthai-context-1.5b-tools	text-generation	504
sakthai-context-0.5b-tools	text-generation	474
sakthai-context-1.5b-merged-v2	text-generation	354
sakthai-vision-7b	vision	337
sakthai-plus-1.5b-lora	text-generation	306
sakthai-plus-1.5b	text-generation	297
sakthai-tts-model	tts	268
sakthai-coder-browser	text-generation	259
sakthai-context-1.5b-tools-v2	text-generation	192
sakthai-coder-1.5b	text-generation	173
sakthai-coder-browser-gguf	text-generation	153
sakthai-coder-browser-lora	text-generation	55
sakthai-embedding	embedding	28
sakthai-plus-1.5b-coder	text-generation	0
sakthai-context-0.5b-tools-sft	text-generation	0
sakthai-context-0.5b-tools-sft-v2	text-generation	0

Companion repos: sakthai-bench-v2 · sakthai-bench-v3

Repository Structure

scripts/
  health-check.py        # Probe inference health across all models
  run-eval.py            # Tool-call evaluation runner
  run-benchmark-v3.py    # Deterministic benchmark generation
  update-model-cards.py  # Refresh README benchmark tables from eval_results
  sakthai-do-all.sh      # One-shot full automation
README.md
.gitattributes

Limitations

• Living toolbox. Script behavior may evolve faster than this card; each script's docstring is the source of truth.
• Network required. Full runs need access to huggingface.co and a valid token.
• CPU-only jobs. GRPO targets CUDA and will not run on free HF Jobs.
• Coverage gaps. Health checks exclude LoRA adapters, config-only recipes, and planned repos without weights.
• Latency not measured. Reports measure availability and benchmark scores, not production latency or SLOs.
• Artifact ownership. Eval results publish to Nanthasit/eval_results; planned consolidation will move artifacts closer to each model.

Outputs & Repo Hygiene

All publish scripts write reports to the top-level Nanthasit/eval_results repo. That repo was created automatically on first publish because --repo-type was not set. Planned consolidation will target per-model .eval_results/ directories or an explicit dataset repo so artifact ownership stays local to each model.

Citation

@misc{sakthai2026,
  title        = {SakThai Pipeline: Zero-Cost Health, Benchmarking, and Card Automation},
  author       = {SakThai Agent Family / beer-sakthai},
  year         = {2026},
  howpublished = {https://huggingface.co/Nanthasit/sakthai-pipeline}
}