Jevlet v6

Jevlet is a small System-One decision model: it reads a state (a typed command plus what is on screen) and a set of typed questions, and returns calibrated probabilities over the options it is given. It never generates text. It is a research replication of the idea behind TypeSafe's Jev, built to drive an always-on Windows command palette where every action is chosen, not written.

• Code, data pipeline, and training notebook: https://github.com/NAME0x0/Jevlet (commit 9d62c46989e9)
• Base encoder: BAAI/bge-small-en-v1.5, 33,508,992 parameters in total
• Skills: 50 (shared/skills.json), each with typed argument questions answered from live options

How it decides

One packed forward pass answers several questions about the same state at once:

Question	Kind	Options come from
Which action does the command ask for?	Choice	the 50 skill names
Is it risky to run without asking?	Noul (yes/no)	fixed
Which app / window / file / reminder …?	Choice	the live environment (installed apps, open windows, stores)
Which part of the command is the text to use?	Choice	spans copied verbatim from the command (shared/text_rules.json)

Branches restart their positions after the shared state (block_bidir topology), so questions do not see each other. A pointer head scores each option at its boundary token. Per-kind temperatures (temperatures in the checkpoint) calibrate choices and yes/no answers separately. The app runs an action on one keypress only when P(safe) ≥ 0.9 and every answer's confidence ≥ 0.75; otherwise it asks.

Results

Reporting corrections (2026-09-26). This card was corrected after release, during the v7 audit (research/v7-audit.md in the GitHub repository). The model weights are unchanged; the v6 tag keeps the original card. What changed:

• grounding is reported per question: the original single figure (78.6%) averaged the control question with a near-trivial risk question;
• training time and loss-log coverage are clarified, with measured and estimated portions distinguished;
• the checkpoint table is labelled as a training progression, not a scaling curve.

Held-out human commands (never used for training or for tuning the span rules)

Test splits of TOPv2, MASSIVE, and CLINC150 mapped to Jevlet's skills (23,279 commands).

Question type	Questions	Accuracy	ECE	ECE (calibrated)
Skill	23,279	98.0%	0.010	0.001
Risk (Noul)	23,279	99.8%	0.030	0.032
Arguments	14,475	97.1%	0.012	0.004

The risk question is trained on soft targets (a risk value per command, for example 0.04 for a benign command). Hard-label ECE scores its confidence against right/wrong, so it penalises a model that reproduces its targets; that is why it rises slightly after calibration. Scored against the soft targets, risk calibration error is 0.0027 before and 0.0013 after temperature scaling, and the mean distance between predicted and target distributions is 0.0064 → 0.0060 (these 23,279 commands, jevlet.metrics.target_calibration_error).

Mixture dev split (all sources)

The calibration temperatures were fitted on the command and real-command dev splits, which are part of this set, so calibrated ECE here is in-sample. The held-out human commands above are not.

Question type	Questions	Accuracy	ECE	ECE (calibrated)
All questions	113,616	97.7%	0.012	n/a
Commands: skill	32,682	99.1%	0.005	0.001
Commands: arguments	29,141	99.2%	0.003	0.001
Grounding	see the per-question breakdown below

Grounding: which on-screen control (reported per question)

Grounding examples come from two apps held out from the grounding training split (Microsoft Teams and Spotify). Each example asks two questions, reported separately below:

• which control to use, where the answer is "None of these controls" when the task belongs to another app (a foreign task);
• the risk yes/no.

The control question is further split by whether the target control is on screen. Accuracy is the argmax against the gold option; ECE (calibrated) uses the checkpoint's per-kind temperature.

Evaluation A: the 3,000 grounding examples in the mixture dev split (the table above).

Question	Slice	Questions	Accuracy	ECE	ECE (calibrated)
Which control	all	3,000	57.3%	0.352	0.321
Which control	target on screen	2,556	50.1%	0.412	0.376
Which control	foreign task (answer: none)	444	98.6%	0.010	0.009
Risk yes/no	all	3,000	100.0%	0.001	0.001

Evaluation B: the full grounding dev set that Evaluation A samples (30,000 examples; the same file, sha256 95acbcaa923c268d27707b8ca8c324a4c0063ad72cb22a272afbdcac5ceb7aa5).

Question	Slice	Questions	Accuracy	ECE	ECE (calibrated)
Which control	all	30,000	56.3%	0.362	0.332
Which control	target on screen	25,418	48.7%	0.426	0.390
Which control	foreign task (answer: none)	4,582	98.5%	0.011	0.009
Risk yes/no	all	30,000	99.96%	<0.001	<0.001

How to read this:

• The overall control accuracy includes foreign tasks, where answering "None of these" is correct and easy (98.5%).
• When the target control is on screen, accuracy is 48.7% (Evaluation B), and the model answers "None of these controls" on 48.6% of those examples (12,341 of 25,418).
• Every one of those answers is wrong, because the target is present. So 94.6% of its errors on target-on-screen examples are false abstentions (12,341 of 13,042 errors), not wrong controls. Evaluation A gives 95.6% (1,220 of 1,276).
• On these held-out apps the model abstains instead of choosing, and does so with high confidence.
• This is v6's weakest capability. Grounding decisions should be confirmed by the user.

Reproduce with python -m scripts.eval_grounding jevlet-v6.pt grounding_v6/dev.jsonl from the GitHub repository (add --only-ids-in mixture_v6/dev.jsonl for Evaluation A). The outputs are in corrections/grounding_eval_A_3k.json and corrections/grounding_eval_B_30k.json.

Training progression on the held-out human commands (one run; not a scaling curve)

These are checkpoints of one cosine-scheduled run, so examples seen and learning-rate annealing are confounded: the final point also benefits from the learning rate decaying to 5%. They show how this run progressed; they do not show how accuracy scales with data. ECE is uncalibrated.

Step	Examples seen	LR factor	Skill accuracy	Skill ECE	Argument accuracy	Risk accuracy
2,500	160,000	0.98	94.9%	0.034	92.3%	99.1%
6,250	400,000	0.88	96.8%	0.019	94.9%	99.5%
12,500	800,000	0.54	97.2%	0.015	96.0%	99.8%
25,000 (final, exported fp16)	1,600,000	0.05	98.0%	0.010	97.1%	99.8%

Calibration temperatures: {'choice': 1.4180543422698975, 'default': 1.1358089447021484, 'noul': 1.0306620597839355}. The hand-written laptop benchmark (jevlet/assistant/benchmark_v2.py, 98 commands, 6 dangerous) runs against the live Windows environment and is reported in the GitHub repository.

Training data

1,350,835 training rows. Nothing personal: the Colab build excludes the author's installed-app list and UI captures. Every row is dropped if it comes within 0.8 token Jaccard (digits collapsed) of a held-out benchmark command.

Source in the mixture	Train rows
commands	600,000
daily	240,000
grounding	150,000
intents	19,647
public	96,222
real	182,674
synthetic	60,000
teacher	2,292

Public data used:

Dataset	Use	License (as reported by the source)
WillHeld/top_v2	TOPv2 (Facebook): human assistant commands, mapped to skills	CC BY-SA 4.0
mteb/amazon_massive_intent	MASSIVE (Amazon), English: intents and mapped commands	CC BY 4.0
clinc/clinc_oos	CLINC150: intents and mapped commands	CC BY 3.0
mteb/banking77	BANKING77: fine-grained intent choice	CC BY 4.0
google/boolq	BoolQ: yes/no reading questions (Noul)	CC BY-SA 3.0
nyu-mll/multi_nli	MultiNLI: entailment as typed questions	mixed; see the dataset card

Synthetic sources are generated by the repository: composed commands with surface variation (typos, text-speak, casing, courtesy words), grounding (which on-screen control), daily decisions, and System-One families (contradiction, missing information, calibration, rules).

Training procedure

Setting	Value
Steps	25,000 at effective batch 64
Learning rate	head 0.002, encoder 0.0001, cosine, warmup 500
Loss	ce_brier (cross-entropy plus Brier, a proper scoring rule)
Precision	bf16 mixed precision
Hardware	NVIDIA A100-SXM4-80GB (Google Colab)
Training time	≈5 h total A100 training (estimated): 3.33 h measured for steps 8,550–25,000 after a resume; first-session duration not recorded.
Throughput	25,960 tokens/s (measured over steps 8,550–25,000)
Peak VRAM	10.2 GB

Logged steps 8,550–25,000. The first training session's progress log was not preserved during the original resume; missing values have not been reconstructed. The resume-logging issue has since been fixed.

progress.jsonl contains the loss curve for steps 8,550–25,000; the first session's log was lost on resume. run_config.json holds the exact configuration and every session's GPU. results.json holds the original evaluation. The corrections above come from scripts/eval_grounding.py (corrections/*.json) and scripts/plot_training.py (figures/training_figures.json).

Files

File	What
jevlet-v6.pt	fp16 checkpoint with per-kind temperatures (loaded by jevlet.system_one.SystemOne)
model.safetensors, config.json	the same weights without pickle, plus model config and temperatures
shared/skills.json, shared/text_rules.json	the skill catalogue and span rules the model was trained against
results.json, progress.jsonl, run_config.json	evaluation, training curve (steps 8,550–25,000), configuration
figures/, corrections/	card figures with their data, and the per-question grounding evaluation (added 2026-09-26)

Use

from huggingface_hub import hf_hub_download
from jevlet.benchmarks import RISK_QUESTION, daily_state
from jevlet.system_one import ChoiceQuestion, NoulQuestion, SystemOne

engine = SystemOne(hf_hub_download("NAME0x0/Jevlet", "jevlet-v6.pt"))
answers = engine.evaluate(
    daily_state("remind me to call the bank at 5", "OUTLOOK: Inbox - Outlook"),
    {
        "skill": ChoiceQuestion("Which action does the command ask for?", ["Set a reminder", "Set an alarm clock", "Search the web"]),
        "risk": NoulQuestion(RISK_QUESTION),
    },
)
print(answers["skill"].selected, answers["skill"].confidence, answers["risk"].probability_true)

The planner in jevlet/assistant/planner.py builds the full question set from the live environment.

Limitations

• English only; tuned for a Windows laptop's apps, windows, and settings pages.
• It chooses among the options it is given. If the right app, file, or span is not offered it cannot answer correctly, and should pick "Not applicable" or ask.
• Free-text arguments are spans of the command. On held-out human commands, span extraction offers the right span for about 85% of reminders, directions, music, and weather requests.
• Arithmetic, dates, and times are parsed by code, not predicted.
• The risk question is a safety net, not a guarantee: destructive actions also have fixed risk floors.