DavidHatley/system-one-mini

System One Mini

System One Mini is an independent research prototype for five fixed, typed decisions over synthetic controlled software-diagnosis summaries. It is a 69.3M-parameter DistilBERT encoder with five depth-2 classification heads.

This is not a reproduction of Jev or RLCD. It is not a chat model, does not generate text, and cannot answer arbitrary questions.

Intended Use

Use this model to study fixed-schema multi-task classification on complete summaries generated under the documented controlled-intervention policy. It supports only:

Decision	Options
task_complete	false, true
should_retry	false, true
failure_source	code, environment, test, dependency, unknown
next_action	inspect_code, rerun_tests, inspect_environment, revert_change, escalate
evidence_sufficient	false, true

Do not use it for arbitrary logs, general coding assistance, natural production incidents, causal claims, safety-critical decisions, or autonomous agent control.

Loading

This repository contains reviewed custom Transformers code. Pin a repository revision in production because trust_remote_code=True executes that code.

from transformers import AutoModel, AutoTokenizer

repo = "DavidHatley/system-one-mini"
tokenizer = AutoTokenizer.from_pretrained(repo)
model = AutoModel.from_pretrained(repo, trust_remote_code=True).eval()

state = "Latest verification: checks failed. Acceptance: not confirmed. ..."
tokens = tokenizer(state, truncation=True, max_length=model.config.max_length, return_tensors="pt")
outputs = model(**tokens)
print([logits.softmax(-1) for logits in outputs.logits])

For named, temperature-scaled outputs, download inference.py from this repository and use its load_model and predict functions.

Architecture And Training

• Base: distilbert/distilbert-base-uncased at revision 12040accade4e8a0f71eabdb258fecc2e7e948be
• Shared encoder followed by five independent Linear -> GELU -> Dropout -> Linear heads
• Maximum input length: 256 tokens
• Full fine-tuning for three epochs in BF16 on one NVIDIA RTX 3060
• 20,000 deterministic synthetic training states; seed 1729
• No reinforcement learning, external teacher model, paid API, or real user traces
• Checkpoint selected by validation mean NLL

Evaluation

The post-selection final set uses one deterministic synthetic renderer written and frozen after candidate selection. It was evaluated once. This tests wording transfer inside the same synthetic policy, not real-world diagnosis.

Final variant	Accuracy	NLL	Brier	ECE	Mean confidence
Raw	86.25%	0.8479	0.2630	0.1273	98.89%
Temperature scaled	86.25%	0.7133	0.3940	0.2673	63.19%

Raw per-head results on the final renderer:

Head	Accuracy	NLL	ECE	Mean confidence
task_complete	100.00%	0.0004	0.0004	99.96%
should_retry	82.65%	1.1987	0.1677	99.24%
failure_source	92.95%	0.2982	0.0451	97.46%
next_action	62.80%	2.4050	0.3641	99.17%
evidence_sufficient	92.85%	0.3373	0.0593	98.63%

The earlier renderer-generalization split was used to compare model candidates and is therefore a development result, not an unbiased final test:

Development variant	Accuracy	NLL	Brier	ECE	Mean confidence
Raw	59.27%	2.4363	0.7666	0.3740	96.58%
Temperature scaled	59.27%	0.7875	0.4331	0.1476	62.39%

Calibration

Per-head scalar temperatures were fitted only on the 2,000-example calibration split. On the final renderer, scaling improved NLL from 0.8479 to 0.7133, but worsened ECE from 0.1273 to 0.2673. Maximum softmax probability must not be interpreted as proven certainty.

Limitations

• Training and evaluation are entirely synthetic.
• The model learns a fixed schema and exact option sets, not arbitrary typed questions.
• Results are from one training seed and one post-selection final renderer.
• Performance is sensitive to wording and representation format.
• Raw predictions are strongly overconfident under distribution shift.
• Temperature scaling did not consistently improve every calibration metric across renderers.
• Inputs longer than 256 tokens are truncated.

Provenance

The model is derived from Apache-2.0-licensed DistilBERT. All added training examples are deterministic project-authored synthetic data. Checkpoint, dataset, calibration, and evaluation hashes are included under provenance/.