D1A-E2B-it QAT · ONNX Runtime (Android / CPU)

What it does: D1A decisions and chat from one ONNX graph and one copy of the weights, on ONNX Runtime's CPU execution provider (Android or any CPU).

Decisions: D1A's adapter is active.
Chat: a null adapter is active, which is exactly Gemma 4 E2B-it QAT.

Text only: this build has no photos, voice or video. Use the MLX build for those.

It is D1A-E2B-it QAT v0.7 (JohnP1/d1a-e2b-it-qat-mlx-q4): the same trunk, adapter, pointer head and temperature (1.9097).

Measured

On a Pixel 10 Pro (Tensor G5), ONNX Runtime 1.22.0, 4 threads, accuracy_level 4 (int8 activations), on a cool phone:


a decision (one call over ~150 new tokens after a cached prefix)	~0.4 s
chat decoding	~12.9 tokens/s at a 1.3k-token past (77 ms/step); faster with a shorter past (56 ms/step at 64)
a chat turn's prefill (150 tokens)	~0.56 s
memory with decisions + chat and a 1.4k-token chat cache	~1.75–1.87 GB PSS

Exactness

The trunk: Google's QAT int4 weights, kept exact as MatMulNBits (block 32). All 276 quantized linear layers are byte-equal to the QAT checkpoint.
The LM head: the MLX build's tied 4-bit table, with an exact correction term instead of float zero points, which take a slow path in ORT 1.22.
accuracy_level 1 (fp32 compute):
- decisions equal the earlier ONNX decision graphs to within 5e-10;
- greedy chat replies are identical to the MLX build on 18 of 22 test prompts. The other 4 split at exact or near ties in MLX's bf16 logits.
accuracy_level 4 (int8 activations):
- decision probabilities move by up to ~0.1;
- greedy chat replies are identical to accuracy_level 1 on 14 of 22 test prompts.
- Act on low-confidence answers only after a confirmation; the int8 flips seen so far were all near ties.

Files

file	what
`d1a-chat-acc4.onnx`	the graph, `accuracy_level` 4 (fast)
`d1a-chat-acc1.onnx`	the same graph, `accuracy_level` 1 (fp32 compute, for parity)
`d1a-trunk.data`	the int4 trunk
`d1a-lmhead.data`	the LM head
`d1a.onnx_adapter`	decisions: the LoRA adapter and pointer head
`d1a-null.onnx_adapter`	chat: rank-0 LoRA with scale 0; the pointer head returns NaN, so a decision run on it fails closed
`embed_q4g64.safetensors`, `ple_q4g64.safetensors`	the token and per-layer embedding tables (4-bit, memory-mapped by the host)
`tokenizer.json`, `encoding-spec.json`	the tokenizer and D1A's request encoding
`d1a_onnx.json`	input and output names for both modes, the chat template, the special and stop ids, and the K/V rule
`reference.py`, `reference_chat.py`	the host procedures in Python: the table gather, D1A's encoding, the chat loop
`goldens.txt`	decision goldens (12 requests from the D1A playground demos)

Use

One session, two RunOptions: one with d1a.onnx_adapter active (decisions), one with d1a-null.onnx_adapter active (chat). The graph's adapter inputs have no defaults, so a chat run needs the null adapter.
Pinned outputs, not IO binding: in ORT 1.22, IO binding drops the active adapter's feeds. Run with preallocated output buffers instead (Java: run(inputs, requestedOutputs, pinnedOutputs, runOptions)).
Decisions:
- D1A's encoding (reference.py), with gen_idx = [];
- a cached state prefix is this graph run with an empty past and a dummy pick at its last token.
Chat:
- render Gemma 4's template with the special ids inserted (d1a_onnx.json), and cache the system prompt's K/V;
- prefill each turn, then decode one token per run: each step's present_* is the next step's past_*;
- stop on <turn|> (106) or <eos> (1). See reference_chat.py.

pip install onnxruntime==1.22.0 tokenizers numpy
python reference_chat.py . "You are a helpful assistant. Answer in one sentence." "" "What is a good way to stay awake?"

Limits

Text only: no photos, voice or video.
Speed: chat decoding is roughly a phone CPU's pace for a 2B model; a long system prompt makes each step slower.
accuracy_level 4 differences: at int8 activations, decisions and chat differ from fp32 compute at near ties (above).
Chat quality: it is Gemma 4 E2B-it QAT's own chat. D1A's training changes only the adapter, which chat switches off.
Requests: questions in English; documents in English or Japanese.
Everything on the D1A-E2B-it QAT card applies.

License and data

Apache-2.0. Base model: Gemma 4 by Google (Apache-2.0). Code: github.com/jonpol01/d1a, built on Kev by Jared Palmer (Apache-2.0). The adapter's training data: intents from MASSIVE 1.1 by Amazon (FitzGerald et al., 2022; CC BY 4.0); Japanese decision data derived from JGLUE by Yahoo Japan Corporation and Waseda University (CC BY-SA 4.0); pull-request data from NousResearch/hermes-agent (MIT), with the labelled PR dataset itself private.