alice-noa-chan/ayaka-v2-small

Ayaka v2 small

Ayaka v2 small is an open, MIT-licensed decision model for Jev-style structured decisions. You give it a state and typed questions (noul / choice / score), and it returns a calibrated probability for every candidate label.

It is a LoRA adapter plus a small decision head on google/gemma-4-E2B-it (text stack only, base revision 3e22461f65e89153144f8adb70e3b8c2cc9845a7). It ships with its frozen serving policy (frozen_tuning.json):

• Noul questions: always reasoned (at most 384 tokens of worked steps, then a re-read).
• Choice and Score questions: reasoned only when a learned router predicts a benefit. On the held-out test the router chose reasoning for none of them.
• All questions: per-path temperatures.

It is trained only on license-clean data.

It did not pass the project's predeclared release gate. It is better than v1 small overall (+8.5 CC), but its Score NLL and Brier are slightly worse than v1's. The gate allows no per-type regression. See Known weakness below. It is published as a separate model with this result stated.

The v1 model, alice-noa-chan/ayaka-small, is unchanged and remains available. The other v2 sizes: ayaka-v2-base (E4B, passed the gate) and ayaka-v2-large (12B).

Code, training pipeline and documentation: https://github.com/alice-noa-chan/ayaka.

What's new in v2

Measured on unseen held-out data, against v1 small:

• +8.5 equal-type CC over v1 small (63.3 vs 54.8, paired 95% CI [+3.7, +13.0]).
• Choice: CC 68.2 → 73.3, NLL 0.637 → 0.550.
• Noul: CC 35.4 → 54.7, NLL 0.410 → 0.282.
• Score: CC 60.6 → 61.9, but NLL 0.898 → 0.919 and Brier 0.494 → 0.499 (slightly worse). See the known weakness below.
• Fewer abstentions: Noul answers inside the 0.2–0.8 band drop from 105 to 66 of 384.

New capabilities v1 does not have (experimental, not part of the measured results above):

• Choice without a full option list. A Choice question can ask the model to propose candidates. The setting goes in the question itself, or in ayaka.questions.<name> when you use the TypeSafe SDK:

  • "candidate_generation": {"mode": "open", "experimental": true, "scope": "..."} with no criteria: the model writes 2–8 mutually exclusive options and keeps a reserved Other residual for anything unlisted.
  • "mode": "expand": splits an existing "other" option of your list into new options.

  Every option, generated or given, is then scored by the decision model.

• Image inputs. serve --images accepts base64 images in the request's ayaka.media field and reads them through Gemma 4's native vision encoder. URLs are never fetched.

  • Text calibration and the router are not applied to image questions.
  • The frozen policy is not applied with --images.
  • Image quality has not been measured for this checkpoint.

Held-out results against v1

The final_test cohort has 836 questions and 431 independent cases. Neither model trained on it, and no fit or choice saw it. It was read once, after the serving policy was frozen. Hardware: one RTX PRO 6000, one request at a time, with the GPU otherwise idle.

system	equal-type CC	Choice CC	Noul CC	Noul abstentions	Score CC	Speed axis	mean latency
v1 ayaka-small (its frozen reasoning policy)	54.8	68.2	35.4	105	60.6	77.4	0.31 s
v2, single pass	60.5	73.3	46.4	88	61.9	89.5	0.07 s
v2 with its frozen policy (default)	63.3	73.3	54.7	66	61.9	72.3	1.28 s

type	NLL v1 → v2	Brier v1 → v2
Choice	0.637 → 0.550	0.328 → 0.282
Noul	0.410 → 0.282	0.113 → 0.076
Score	0.898 → 0.919	0.494 → 0.499

Known weakness: Score calibration

v2 small is not better calibrated than v1 small on Score questions.

• Score CC is 1.3 points higher than v1's.
• Score NLL is 0.021 worse and Brier 0.005 worse, over 192 questions.
• The frozen policy answers every Score question in a single pass, so these numbers are the single-pass model's.

The project predeclared a release gate before this run: no per-type regression against v1, with zero tolerance. v2 small failed that gate on the Score NLL and Brier checks, and was therefore not adopted as a replacement for v1 small. Every other check passed:

• CC gain of at least 5, with a paired 95% interval above zero;
• Choice and Noul per-type checks;
• clustered subgroups;
• Speed axis 72.3.

If your workload is mostly rubric scoring, compare the two models on your own data, or use ayaka-v2-base, which passed the gate.

A second, report-only comparison found that always reasoning on Noul lowers Noul CC on the Japanese and Korean MASSIVE questions, compared with v2's own single pass. The full report is docs/experiments/V2_FAMILY_RUN_RESULT_2026-10-11.md in the repository.

JevBench public tiers have not been measured for v2.

Use (pinned)

pip install -c https://raw.githubusercontent.com/alice-noa-chan/ayaka/0a97b5e28fbbe3290866b03391d43f7cce9b40e0/constraints.txt \
  "git+https://github.com/alice-noa-chan/ayaka@0a97b5e28fbbe3290866b03391d43f7cce9b40e0"
python -m ayaka.serve --ckpt alice-noa-chan/ayaka-v2-small --device cuda --port 8000

The server finds frozen_tuning.json in the repository, loads the adapter merged, and applies the frozen policy, router and temperatures. These are the settings the held-out numbers above were measured with.

• Pinned code and the measured code: the pinned code adds one fix after the measured run (0a97b5e).

  • Before the fix: when a long question overflowed Gemma 4 E2B's 512-token sliding-window cache, the reasoned read fell back to the single-pass answer. This happened on 11 of 1,336 tuning questions and on none of final_test.
  • With the fix: such questions are read in full instead.

• Single pass: add --frozen-tuning off --reasoning-mode off. Every question then takes one forward pass, which is faster (Speed axis 89.5, mean 0.07 s) but less accurate on Noul.

• API: the server speaks TypeSafe's POST /v1/systemone format and returns a probability for every label:

  • noul: P(true);
  • choice: per-label probabilities;
  • score: per-level probabilities and the expected score.

  usage.input_tokens and usage.output_tokens are reported per request.

• Files: this repository holds the adapter (fp32 safetensors, 193 MB), the head (head.safetensors), the config, the Noul calibration and the frozen policy. The base model is fetched from Google's repository at the pinned revision.

Naming. electra_config.json is a legacy alias of ayaka_config.json; both hold the same config. adapter/adapter_config.json records the path of the base model on the training machine. It is left byte-identical because the Noul calibration is bound to its hash. Ayaka always loads the base from ayaka_config.json.

Training

• Initialization: a fresh LoRA on the pinned base (not continued from v1). Gold labels only, with no distillation and no teacher outputs.

• Adapter: LoRA r 32, alpha 64, dropout 0.05, on q/k/v/o and gate/up/down projections. Decision head: pointer 256, 2 mixer layers, 4 heads.

• Schedule: 2 epochs of 37,040 rows, 32 rows per step (2,315 scheduled steps), cosine with 3% warmup. LoRA LR 3e-5, head LR 1.5e-4. This is the v2 large recipe, with only the size settings of v1 small changed.

• Losses: NLL 1.0, Brier 0.5, RPS 0.35, missing-label 0.25.

• Checkpoint selection: every 250 steps on a reserved split, with early stopping after three reads without improvement. Training stopped at step 2,000; the released weights are the step-1,250 state.

• Serving fit:

  • per-path temperatures fitted on a reserved calibration cohort;
  • the router fitted on router_train and promoted on dev;
  • the policy chosen on router_train.

  All of it was frozen and hash-sealed before the final_test read.

• Hardware: one RTX PRO 6000 (96 GB), shared with the v2 base run. The run for both sizes took 5.5 h.

Training data

No commercial-LLM outputs and no Jev API labels were used. Held-out cohort items were excluded by source group, lineage and 13-word overlap.

source	dataset	licence	train rows
helpsteer2	nvidia/HelpSteer2	CC BY 4.0	3,000 responses (15,000 Score questions)
commonsense_qa	tau/commonsense_qa	MIT	6,400
massive_ko	AmazonScience/massive (ko-KR)	CC BY 4.0	3,000 utterances
massive_ja	AmazonScience/massive (ja-JP)	CC BY 4.0	3,000 utterances
contract_nli	ContractNLI	CC BY 4.0	108 documents
strategyqa	ChilleD/StrategyQA	MIT	1,036
authored	generated and verified by this repository	MIT	256 per type

CC BY sources were used as training data only. Their licence and attribution are listed here.

Benchmark disclosure

• Non-public data only: training, checkpoint selection, calibration, router fitting, policy choice and the release gate all used non-public held-out cohorts drawn from validation and test splits.
• No JevBench in training: no JevBench items or labels are in the training corpus.
• No public results in this release: public JevBench results were not used for any choice in it.

License

MIT for code and fine-tuned weights. The base model google/gemma-4-E2B-it is Apache-2.0, and its notices apply to the base portion.