alice-noa-chan/ayaka-v2-base

Ayaka v2 base

Ayaka v2 base is an open, MIT-licensed decision model for Jev-style structured decisions. You give it a state and typed questions (noul / choice / score), and it returns a calibrated probability for every candidate label.

It is a LoRA adapter plus a small decision head on google/gemma-4-E4B-it (text stack only, base revision ee0ef6023621cff504d758262d4e04895a5af4a2). It ships with its frozen serving policy (frozen_tuning.json):

• Noul questions: always reasoned (at most 384 tokens of worked steps, then a re-read).
• Choice and Score questions: reasoned only when a learned router predicts a benefit.
• All questions: per-path temperatures.

It is trained only on license-clean data.

It passed the project's predeclared release gate against v1 base, including the zero-tolerance per-type checks.

The v1 model, alice-noa-chan/ayaka-base, is unchanged and remains available. The other v2 sizes: ayaka-v2-large (12B) and ayaka-v2-small (E2B).

Code, training pipeline and documentation: https://github.com/alice-noa-chan/ayaka.

What's new in v2

Measured on unseen held-out data, against v1 base:

• +15.1 equal-type CC over v1 base (78.1 vs 63.0, paired 95% CI [+11.1, +19.0]).
• Choice: CC 67.2 → 75.9, NLL 0.673 → 0.502.
• Noul: CC 60.4 → 89.6, NLL 0.281 → 0.153.
• Score: CC 61.3 → 68.9, NLL 0.949 → 0.787.
• Far fewer abstentions: Noul answers inside the 0.2–0.8 band drop from 59 to 7 of 384.
• Learned reasoning router: v1 reasons only on calculation questions picked by fixed rules. v2 always reasons on Noul and uses a router, trained and promoted on held-out data, for Choice and Score.

New capabilities v1 does not have (experimental, not part of the measured results above):

• Choice without a full option list. A Choice question can ask the model to propose candidates. The setting goes in the question itself, or in ayaka.questions.<name> when you use the TypeSafe SDK:

  • "candidate_generation": {"mode": "open", "experimental": true, "scope": "..."} with no criteria: the model writes 2–8 mutually exclusive options and keeps a reserved Other residual for anything unlisted.
  • "mode": "expand": splits an existing "other" option of your list into new options.

  Every option, generated or given, is then scored by the decision model.

• Image inputs. serve --images accepts base64 images in the request's ayaka.media field and reads them through Gemma 4's native vision encoder. URLs are never fetched.

  • Text calibration and the router are not applied to image questions.
  • The frozen policy is not applied with --images.
  • Image quality has not been measured for this checkpoint.

Held-out results against v1

The final_test cohort has 836 questions and 431 independent cases. Neither model trained on it, and no fit or choice saw it. It was read once, after the serving policy was frozen. Hardware: one RTX PRO 6000, one request at a time, with the GPU otherwise idle.

system	equal-type CC	Choice CC	Noul CC	Noul abstentions	Score CC	Speed axis	mean latency
v1 ayaka-base (its frozen reasoning policy)	63.0	67.2	60.4	59	61.3	86.1	0.38 s
v2, single pass	70.4	73.3	70.8	39	67.1	88.0	0.10 s
v2 with its frozen policy (default)	78.1	75.9	89.6	7	68.9	64.0	2.11 s

NLL and Brier also improve over v1 for every type:

type	NLL v1 → v2	Brier v1 → v2
Choice	0.673 → 0.502	0.351 → 0.254
Noul	0.281 → 0.153	0.078 → 0.036
Score	0.949 → 0.787	0.514 → 0.436

The project predeclared a release gate before this run. v2 base passes all of it:

• CC gain of at least 5, with a paired 95% interval above zero;
• at least 200 independent cases;
• no per-type regression against v1, with zero tolerance;
• clustered per-source and per-language non-inferiority;
• Speed axis of at least 50.

The full report is docs/experiments/V2_FAMILY_RUN_RESULT_2026-10-11.md in the repository.

JevBench public tiers have not been measured for v2.

Use (pinned)

pip install -c https://raw.githubusercontent.com/alice-noa-chan/ayaka/0a97b5e28fbbe3290866b03391d43f7cce9b40e0/constraints.txt \
  "git+https://github.com/alice-noa-chan/ayaka@0a97b5e28fbbe3290866b03391d43f7cce9b40e0"
python -m ayaka.serve --ckpt alice-noa-chan/ayaka-v2-base --device cuda --port 8000

The server finds frozen_tuning.json in the repository, loads the adapter merged, and applies the frozen policy, router and temperatures. These are the settings the held-out numbers above were measured with.

• Pinned code and the measured code: the pinned code adds one fix after the measured run (0a97b5e).

  • Before the fix: when a long question overflowed Gemma 4 E4B's 512-token sliding-window cache, the reasoned read fell back to the single-pass answer. This happened on 4 of 1,336 tuning questions and on none of final_test.
  • With the fix: such questions are read in full instead.

• Single pass: add --frozen-tuning off --reasoning-mode off. Every question then takes one forward pass, which is faster (Speed axis 88.0, mean 0.10 s) but less accurate.

• API: the server speaks TypeSafe's POST /v1/systemone format and returns a probability for every label:

  • noul: P(true);
  • choice: per-label probabilities;
  • score: per-level probabilities and the expected score.

  usage.input_tokens and usage.output_tokens are reported per request.

• Files: this repository holds the adapter (fp32 safetensors, 279 MB), the head (head.safetensors), the config, the Noul calibration and the frozen policy. The base model is fetched from Google's repository at the pinned revision.

Naming. electra_config.json is a legacy alias of ayaka_config.json; both hold the same config. adapter/adapter_config.json records the path of the base model on the training machine. It is left byte-identical because the Noul calibration is bound to its hash. Ayaka always loads the base from ayaka_config.json.

Training

• Initialization: a fresh LoRA on the pinned base (not continued from v1). Gold labels only, with no distillation and no teacher outputs.

• Adapter: LoRA r 32, alpha 64, dropout 0.05, on q/k/v/o and gate/up/down projections. Decision head: pointer 384, 2 mixer layers, 6 heads.

• Schedule: 2 epochs of 37,040 rows, 32 rows per step (2,315 scheduled steps), cosine with 3% warmup. LoRA LR 3e-5, head LR 1.5e-4. This is the v2 large recipe, with only the size settings of v1 base changed.

• Losses: NLL 1.0, Brier 0.5, RPS 0.35, missing-label 0.25.

• Checkpoint selection: every 250 steps on a reserved split, with early stopping after three reads without improvement. Training stopped at step 1,750; the released weights are the step-1,000 state.

• Serving fit:

  • per-path temperatures fitted on a reserved calibration cohort;
  • the router fitted on router_train and promoted on dev;
  • the policy chosen on router_train.

  All of it was frozen and hash-sealed before the final_test read.

• Hardware: one RTX PRO 6000 (96 GB), shared with the v2 small run. The run for both sizes took 5.5 h.

Training data

No commercial-LLM outputs and no Jev API labels were used. Held-out cohort items were excluded by source group, lineage and 13-word overlap.

source	dataset	licence	train rows
helpsteer2	nvidia/HelpSteer2	CC BY 4.0	3,000 responses (15,000 Score questions)
commonsense_qa	tau/commonsense_qa	MIT	6,400
massive_ko	AmazonScience/massive (ko-KR)	CC BY 4.0	3,000 utterances
massive_ja	AmazonScience/massive (ja-JP)	CC BY 4.0	3,000 utterances
contract_nli	ContractNLI	CC BY 4.0	108 documents
strategyqa	ChilleD/StrategyQA	MIT	1,036
authored	generated and verified by this repository	MIT	256 per type

CC BY sources were used as training data only. Their licence and attribution are listed here.

Benchmark disclosure

• Non-public data only: training, checkpoint selection, calibration, router fitting, policy choice and the release gate all used non-public held-out cohorts drawn from validation and test splits.
• No JevBench in training: no JevBench items or labels are in the training corpus.
• No public results in this release: public JevBench results were not used for any choice in it.

License

MIT for code and fine-tuned weights. The base model google/gemma-4-E4B-it is Apache-2.0, and its notices apply to the base portion.