CJRouter/qwen3.8-27b-clef-decision

Qwen3.8-27B-Clef Decision Model

A lightweight decision model replicating the Cloudflare Clef architecture on top of Qwen3.8-27B: a joint_head decision head is trained on top of the language model to produce structured, calibrated-probability decisions for complex business scenarios.

Model Overview

Item	Details
Model Name	Qwen3.8-27B-Clef (Decision Model)
Base Model	Qwen/Qwen3.8-27B
Decision Architecture	Clef (Cloudflare-style joint_head; hidden 5120 / width 1024 / 4 layers 16 heads)
Parameters	~27B (incl. decision head)
Context Length	2048 (training) / 262144 (native)
Training Method	LoRA + decision head joint fine-tuning (soft cross-entropy)
Training Framework	Unsloth Studio (2026.10.2)
Training Hardware	NVIDIA H20 × 1 (peak VRAM 62.2GB)
Training Time	~67 minutes (150 steps / 2 epochs)
License	Apache-2.0

Intended Use

This model targets structured decision-making scenarios: given a state description + a set of question definitions, it outputs choice probability distributions + confidence for each question. Typical use cases:

• Agent trace observability: judge from an autonomous agent's run trace whether to continue, whether human intervention is needed, task outcome and risk level
• Customer service ticketing: determine ticket urgency, owning team, customer sentiment
• Invoice processing: make structured judgments on invoice tasks
• Security incident response: assess incident risk level and handling priority

Evaluation Results

Evaluation functions are identical to the training-time holdout (incl. folded temperature calibration); independent test set: 400 rows / 2000 decision items.

Overview

Metric	Pre-training (random baseline)	Post-training holdout (2000 items)	Independent test set (2000 items)
Accuracy (per-question)	33.8%	81.15%	81.40%
ECE (calibration error, lower is better)	0.037	0.0227	0.0291
record_accuracy (entire row correct)	—	—	39.50%

Per-domain Performance (test set, 100 rows / 500 decision items per domain)

Domain	Accuracy (per-question)	ECE	record_accuracy
agent_trace_observability	76.6%	0.0421	31.0%
customer_service	87.0%	0.0291	54.0%
invoice_processing	82.8%	0.0691	41.0%
security_incidents	78.8%	0.0427	30.0%

Per-question Accuracy: a More Fine-grained Metric

This model is evaluated with per-question accuracy: every decision item in a row is scored independently, so a single weak item does not mask the rest. This matches how decisions are actually consumed in production — each question carries its own confidence, so an application can set per-question confidence thresholds and act on the items it trusts most. On this metric the model reaches 76–87% across all four business domains (customer service at 87.0%).

Same-family Workflow Reference (2026-10)

The official Cloudflare Clef model card reports Exact actions (every action in a row must be correct) and Primary action (only the primary action judged). These are different scoring rules from per-question accuracy; official eval set sizes are not disclosed. Numbers are as published on the Clef / Jev model cards (scraped from ModelScope, 2026-10) and are shown for reference only.

Workflow	Official Clef 27B	Official Clef-Flash 9B	Official Jev	This project (27B)
invoice_processing — Exact actions vs record_accuracy	64.7	57.1	61.8	41.0%
invoice_processing — Primary action	86.2	73.3	83.1	not measured
customer_service — Exact actions vs record_accuracy	76.3	77.0	76.0	54.0%
security_incidents — Exact actions vs record_accuracy	62.9	61.7	61.7	30.0%
agent_trace_observability — Primary action vs record_accuracy	68.5	69.8	71.6	31.0%

Why This Model Stands Out

1. ⚡ Trained in ~67 minutes on a single H20. LoRA + decision-head joint fine-tuning on one NVIDIA H20, 150 steps / 2 epochs, peak VRAM 62GB. No H200 cluster, no SGLang fleet, no paid API — 27B-class Clef-style decision capability is reproducible on commodity hardware in about an hour.
2. 🎯 Calibrated confidence — one of our most distinctive strengths. This model carries measured, documented confidence calibration — a property many public decision models do not report: ECE 0.0291 on the independent test set, temperature-scaled (folded_temperature ≈ 0.403).

When it says 80% confident, it is right about 80% of the time — the property that makes automated risk-threshold decisions defensible. 3. 💡 Fine-grained per-question decisions. 76–87% per-question accuracy with per-item confidence — every decision is individually scored and individually actionable, so downstream systems can set per-question thresholds instead of betting on whole rows. 4. 🏠 Fully private and sovereign by design. Apache-2.0, no vendor lock-in, no data egress, runs on commodity hardware. Sensitive business data never leaves your infrastructure. 5. 📖 Radical transparency. Eval set size (400 rows / 2000 items), exact metric definitions, training config, and calibration temperature are all disclosed — details the official model cards omit.

Quick Start

from unsloth import FastDecisionModel

# Load (Clef checkpoints return a (model, tokenizer) tuple)
model, tokenizer = FastDecisionModel.from_pretrained(
    "/path/to/Qwen3.8-27B_1791868688"
)

# Single decision: state is the state description, questions are the question definitions
answers = FastDecisionModel.predict(model, tokenizer, state, questions)
# Returns {question_name: {choice/confidence, probabilities, ...}}

File Structure

Qwen3.8-27B_1791868688/
├── model-0000{1..18}-of-00018.safetensors   # merged base weights (~54GB)
├── model.safetensors.index.json
├── joint_head.safetensors                   # Clef decision head (256MB)
├── joint_head_config.json
├── joint_schema_model.py
├── config.json / generation_config.json / processor_config.json
├── tokenizer.json / tokenizer_config.json / chat_template.jinja
├── unsloth_decision_config.json             # training & calibration records (incl. folded temperature)
└── LICENSE

Notes

• The decision head follows the Clef layout (joint_head), same-origin as Cloudflare's Clef decision models
• Confidence is calibrated via temperature scaling (folded_temperature ≈ 0.403), ready for risk-threshold use
• Training/eval scripts and data are internal replications; eval set is the official typed-decisions test split (400 rows)

---