AndyInQtr/laya-decision-plugin

Laya Decision-Plugin — combined Core AI decision model (r15)

A typed decision classifier for coding agents: given an agent state and a typed question, it answers which tool, which skill, allow/ask/block, language, reply-or-act — with a calibrated confidence and an act/escalate readout. Zero token generation. One .aimodel asset (Apple Core AI, f16, bounded-dynamic shapes) ships a shared frozen encoder plus five head-chains selected per sample (head_idx): base triage, lang_route, guardrail (PPO), tool_route (PPO), skill_route (PPO). Runs on GPU/ANE/CPU via coreai-core — 5–6 ms per pass on Apple Silicon (macOS 27+).

This is an open-source alternative in the "decision-head agent plugin" category: it does not replace the main LLM — it is the fast, on-device decision layer in front of it (route, guard, escalate).

Scores (frozen 239-row session-disjoint test, floor 0.7)

Use case	Head	Base → Final	Protocol	Verdict
triage	base	0.875	real test	GO
lang_route	chain 1	0.159 → 0.958	slate holdout	GO
tool_route (20 tools)	chain 3	0.000 → 0.750 @ 80% coverage	frozen oc2 n=45, toolace-ho 0.919	GO
skill_route (107 skills)	chain 4	0.242 → 0.615 @ 100% coverage	n=200 dual-blind holdout	GO
guardrail (allow/ask/block)	chain 2	red 10/35 → red 2 seen / 1 unseen; acc 0.465	two never-trained adversarial oracles	GO (model-primary)
act/escalate (rm -rf class)	chain 2	acts-on-everything → stops 53/55 seen, 29/32 unseen	same oracles	GO as gate (NOT frictionless autonomy)

Deployment ECE: 0.188 (test) / 0.135 (open holdout). Full evidence: eval/*.json (every artifact must report cases==239), ledgers in docs/.

What ships here

• laya-combined-f16.aimodel/ — the single combined asset (B=1, L≤1024, K≤128).
• combined_provenance.json — sha256s of pinned source + every head, torch parity numbers (maxabs 0 on 96 real rows), shape.
• configs/ — per-head fitted deployment temperatures (logits ship un-tempered; head weights are NOT redistributed — rebuild via the recipe, their sha256s are pinned in combined_provenance.json).
• datasets/ft/ — split_manifest.json (bucket tallies — provenance structure only, no rows) + the filter note. No split rows ship (train/val/test/open_holdout are local-session harvests containing private paths; the 239-row test is bit-frozen here and every per-case score ships in eval/*.json). Rebuild with scripts/reproduce_recipe.sh — see docs/REPRODUCE.md. Synthesis and public-open corpora (dataset_synth*.jsonl, dataset_open_*.jsonl, adversarial_guard*.jsonl) ship in full; personal-harvest files (dataset.jsonl, dataset_oc, dataset_all*, dataset_pi*) are NOT redistributed (private paths/emails in harvested text — same regen path).
• eval/ — every scored artifact, ceilings, routed deployment table.
• scripts/reproduce_recipe.sh — one command: harvest → split (leak-gated) → SFT (warm-start) → PPO (class-balanced, oracles) → calibration → combined asset export (sha + parity gated). docs/REPRODUCE.md explains each stage.

How to run

# pip install coreai-core transformers numpy  (no torch, no Xcode needed)
from laya_port.combined_agent import CombinedAgent
ag = CombinedAgent("laya-combined-f16.aimodel", "configs", unit="gpu")
d = ag.decide("guardrail", state="rm -rf /home/user/projects",
              question={"disposition": {"type": "choice", "instructions": "...",
                                        "criteria": {"allow": "...", "block": "..."}}})
# -> {'choice': 'block', 'confidence': 0.97, 'acted': True, ...}

Pin the compute unit (gpu default; unpinned loads can SIGABRT on ANE type-inference). One CombinedAgent per process; reuse it.

Training in one paragraph

Head-only fine-tuning (encoder frozen — verified bit-identical across heads, which is what makes the combined asset legal), warm-start continuation (warm > fresh at this scale, ablation in docs/GO_GATE.md), PPO over the decision for safety heads (reward −12 catastrophic; class-balanced selection set — the single most load-bearing safety knob), dual-blind teacher verification of every label, session-disjoint splits with machine-re-asserted leak flags. 15 rounds, every VOID round and incident documented in docs/FINETUNE.md — including a test-leak caught by the pipeline's own assertions with a deterministic no-retrain proof.

Limitations (honest)

• Golds are stronger-teacher agreement, not human consensus. Teacher self-agreement ceilings (0.38–0.67 per corpus) are measured and shipped.
• Skill-route real in-harness traffic is ~16 rows; the verdict rests on a dual-blind-graded public-sketch holdout.
• Guardrail unseen-red band is 1–3 across PPO seeds — treat it as a confident gate with a regex advisory backstop (src/laya_port/safety_gate.py, 33 tests), not as a sole destructive-action防线. Do not use this model as frictionless autonomy for destructive classes.
• Trained for the 20-tool / 107-slate coding-agent harness; foreign tool slates degrade gracefully to escalation, not to correct guesses.
• Base checkpoint convaiinnovations/laya-multilingual is Apache-2.0; derivative weights here inherit Apache-2.0. Public corpora licenses are listed in docs/RESEARCH_SUMMARY.md (pi traces: "other" — reference only).

Built on macOS 27.2 / Apple Silicon (M5 Max) with coreai-torch 0.4.2 + torch 2.13.0. Questions → docs/ first; everything measurable is in there.