KnowLine-26B-A4B-Gen1

The first release of PelaAI's System One decision model on a Gemma 4 base: a 26B-A4B mixture of experts, trained in the same agent-driven data loop as KnowLine-35B-A3B and the KnowLine-4B line.

Inference guide · 中文说明 · weights Apache-2.0 · siblings: KnowLine-35B-A3B-Gen1, KnowLine-4B-Gen4

Highlights:

• Decision Index 0.3 public suite: 65.76 (self-run). KnowLine-35B-A3B-Gen1 scores 69.49 on the same suite and 62.92 on the board's full score; see Comparison.

• Stronger on decision tasks from new domains: on our larger set of decision tasks from domains not in training (34 tasks) it scores 59.9, against 52.7 for KnowLine-35B-A3B-Gen1 and 44.5 for the stock gemma-4-26B-A4B-it. On our older 10-task set it scores 55.2 (35B-A3B-Gen1: 56.4).

• Harder to steer with planted instructions: on our injection set, 0.3% of answers changed because of an instruction planted in the state; with injection templates not seen in training, 1.8% did (stock gemma-4-26B-A4B-it: 60.9% and 56.8%; 35B-A3B-Gen1: 0.0% and 0.8%).

• Calibrated probabilities: a temperature is folded into the weights. Computed with the board's method, ECE is about 0.02-0.03 (board median: 0.078) and the Brier score about 0.27-0.29 (board median: 0.46); see Calibration.

• Our other evaluations: English in-house set 70.6 (35B-A3B-Gen1: 70.3), JevBench / JevBench-hard 89.6 / 80.2 (85.7 / 70.3). It is clearly lower than the 35B-A3B on Chinese: C-Eval 74.3 (87.6) and our Chinese in-house sets 72.2 / 69.2 (73.8 / 72.1); and on Open-Jev 1.1 test / OOD: 83.2 / 85.0 (91.0 / 90.4); see Evaluation.

• Images without any image training: trained on text only, it keeps the base model's vision tower. On our reconstruction of the Decision Index 0.3.1 vision public suite it scores 64.9, against 59.7 for the stock gemma-4-26B-A4B-it on the same harness (35B-A3B-Gen1: 71.2); see Images.

• KOF '98 harness: 15-2-1 against Jev, 17-1-0 against StartLux-Decision-35B-A3B and 8-9-1 against KnowLine-35B-A3B-Gen1 in single-bout mirror matches; see Game harness.

• An AI agent ran the whole loop end to end. In each round it:

  • identified where the model was weak;
  • proposed a targeted data group;
  • built and decontaminated the data;
  • trained and evaluated the model on it;
  • kept or rejected the change based on the evidence.

  For this model, see How Gen1 was made.

KnowLine is an independent model. It is not affiliated with or endorsed by TypeSafe, Jev or Google, and it is not derived from TypeSafe or Jev.

What it does

You send a state (text or a chat) and up to 64 typed questions: yes/no, choose one of k, or score on a rubric. The model answers every question with a probability distribution over its options:

• one forward pass per question, with no text generation;
• the output is the probability of each option's label token;
• existing Jev clients only need to change the base URL, and users of other KnowLine models only need to change the model name, because the interface is the same.

Base model	google/gemma-4-26B-A4B-it (Apache-2.0): mixture of experts, about 26B parameters in total and about 4B active per token
Training	LoRA SFT (rank 32, alpha 64, language model only, including the routed experts), merged into bf16 weights; then a short second LoRA round, also merged
Release format	bf16 weights (about 52 GB), with a readout temperature T = 1.25 folded into the final norm weight and the final logit soft-cap (see Calibration). The vision tower is unchanged from the base model; the tokenizer is Gemma 4's, not the one of the Qwen-based KnowLine models.
Serving	serve_knowline.sh: SGLang in bf16. We do not convert the weights to FP8 at load for this model: in our tests two FP8 runs of the same questions agreed on only 95.7% of the decisions, against 99.4% for bf16. SGLang reserves 85% of the GPU's memory by default (MEM; about 82-88 GB on a 96 GB card); the weights take about 52 GB, so one 80 GB GPU is enough. The transformers backend peaks at about 60 GB. We ran every evaluation with one server on one 96 GB H20.
Languages	English, Simplified Chinese, Traditional Chinese

Quickstart

pip install "sglang==0.5.21" "transformers==5.12.1" requests
bash serve_knowline.sh PelaAI/KnowLine-26B-A4B-Gen1 0 8080     # SGLang (bf16) on :9080 + /v1/systemone on :8080
curl -s http://127.0.0.1:8080/v1/systemone -H 'Content-Type: application/json' -d '{
  "model": "m",
  "state": "Customer: my order arrived broken, I want my money back.",
  "questions": {"refund": {"type": "noul", "instructions": "Should the agent offer a refund?"},
                "tone": {"type": "choice", "instructions": "Customer tone?",
                         "criteria": {"angry": "Angry", "neutral": "Neutral", "happy": "Happy"}}}}'

• Front end: knowline_server.py is a single file that depends only on transformers and requests. It is the same code as in the other KnowLine releases; it renders each question with the model's own chat template.
• Full settings: the exact settings used for our Decision Index run are in INFERENCE.md.

How Gen1 was made

1. Why a Gemma base? KnowLine-35B-A3B-Gen1 is strong on the public suite and on private tests of the same skills, but on decision tasks from new domains it trails models of similar size. On our own new-domain tasks, the stock gemma-4-26B-A4B-it already reads such tasks better than the stock Qwen3.5-35B-A3B. The agent trained the latest data round of the 35B line on this base to see how much of the new-domain gap comes from the base model.

2. Data. Two rounds:

   • the latest data round of the 35B line, rendered with Gemma's chat template;
   • a short second round of injection-invariance items (instructions planted in the state that must not change the answer), plus a replay share of the first round.

   All data went through the same decontamination check as the earlier releases; see Disclosures.

3. Training. One epoch on the first round, then one epoch on the second round, each with a LoRA adapter merged into the weights. A single temperature was fitted and folded into the weights. It changes the size of the probabilities, but not which option ranks first.

4. Selection. The first-round model passed our checks against the stock gemma-4-26B-A4B-it except one: planted instructions changed 6.6% of its answers, above our limit of 4%. The second round brought this to 0.3% (unseen templates: 16.2% → 1.8%) without a significant change on our held-out sets or Open-Jev 1.1 OOD, within our margin on new-domain tasks (10-task set 56.4 → 55.2; 34-task set 59.3 → 59.9), and with a significant gain on JevBench-hard (73.9 → 80.2), so it is the release. Against KnowLine-35B-A3B-Gen1, it is level on our English in-house set and higher on JevBench-hard, but significantly lower on our Chinese in-house sets and on Open-Jev 1.1 OOD.

Comparison

Scores on the Decision Index 0.3 board. The full score also includes private tests that only the maintainers run (weighted 0.5 same-skill and 0.3 new-domain); the full score of this model is not available yet.

model	size	DI 0.3 public	DI 0.3 full	source
KnowLine-26B-A4B-Gen1	26B-A4B	65.76	not yet scored	self-run, official kit
KnowLine-35B-A3B-Gen1	35B-A3B	69.49	62.92	board
StartLux-Decision v1.1 27B	27B	72.88	69.12	board
linne-oak 1.0	31B	63.94	65.61	board
span-02	26B-A4B	63.62	62.56	board
Shisa DE-2 (repeat)	26B-A4B	57.57	59.77	board
Jev 1.14	(API)	60.70	61.53	board
AJev lora1	26B-A4B	57.67	56.43	board
GEV-26B-Decide	26B-A4B	58.03	56.17	board

Board rows as of the board's latest update (2026-10-16).

Public-suite scores by area (DI 0.3):

area	KnowLine-26B-A4B-Gen1	KnowLine-35B-A3B-Gen1
Knowledge & reasoning	50.4	52.5
Language	69.2	71.2
Retrieval & routing	69.7	75.2
Tools & agents	82.2	88.1
Arts & taste	58.6	63.4

Releases

The model is trained in a self-evolving loop, and new versions will follow.

model	date	DI 0.3 public	golden held-out (en / zh-Hans / zh-Hant)	notes
KnowLine-26B-A4B-Gen1 (this model)	2026-10-19	65.76	70.6 / 72.2 / 69.2	first release on a Gemma 4 base
KnowLine-35B-A3B-Gen1	2026-10-12	69.49	70.3 / 73.8 / 72.1	first 35B-A3B release; DI 0.3 full 62.92
KnowLine-4B-Gen4	2026-10-10	64.90	69.7 / 70.1 / 65.1	4B, fourth release
KnowLine-4B-Gen3	2026-10-09	63.11	69.4 / 70.8 / 65.0	4B, third release
KnowLine-4B-Gen2	2026-10-08	62.54	69.7 / 71.2 / 64.8	4B, second release
KnowLine-4B-Gen1	2026-10-07	60.47	68.7 / 69.9 / 64.1	4B, first release

Evaluation

All results are self-run and have not been verified by a third party.

Held-out and Chinese evaluations

suite	26B-A4B-Gen1	35B-A3B-Gen1	stock gemma-4-26B-A4B-it
In-house evaluation set, English	70.6	70.3	66.9
In-house evaluation set, Simplified Chinese	72.2	73.8	71.8
In-house evaluation set, Traditional Chinese	69.2	72.1	68.3
C-Eval (4 categories, macro)	74.3	87.6	67.5
Open-Jev 1.1 test / OOD	83.2 / 85.0	91.0 / 90.4	72.0 / 76.8
JevBench / JevBench-hard	89.6 / 80.2	85.7 / 70.3	89.2 / 79.3
Decision tasks from new domains (our own, 34 tasks)	59.9	52.7	44.5
Decision tasks from new domains (our own, 10 tasks)	55.2	56.4	55.9
Prompt injection: answers changed (lower is better)	0.3%	0.0%	60.9%
Prompt injection with templates not seen in training (lower is better)	1.8%	0.8%	56.8%

In a paired sign test against 35B-A3B-Gen1, the drops on the Simplified and Traditional Chinese in-house sets (p = 0.011 and p = 0.008) and on Open-Jev 1.1 OOD (p < 0.001) are significant, and so is the gain on JevBench-hard (p = 0.013); the difference on the English in-house set is not (p = 0.60). Against the stock gemma-4-26B-A4B-it, the training raised C-Eval, Open-Jev 1.1 and our in-house sets and cut the injection rate from 60.9% to 0.3%.

Web operation (Mind2Web official test splits, evaluation only)

split	26B-A4B-Gen1 element selection	26B-A4B-Gen1 operation (balanced)	35B-A3B-Gen1 element selection	35B-A3B-Gen1 operation (balanced)
test_task (websites seen in training, new tasks)	87.1	92.2	94.4	97.4
test_website (new websites)	87.0	93.8	92.4	97.5
test_domain (new domains)	87.0	93.9	93.1	98.0

• Lower than 35B-A3B-Gen1 on all three splits.
• The task is to pick the target element from a candidate set: the target plus up to 5 other elements sampled from the page. This is easier than the original Mind2Web protocol, so do not compare these results with the Mind2Web leaderboard.

Images (zero-shot)

• No image training: Gen1 keeps the base model's vision tower unchanged and saw no images in training. knowline_server.py accepts OpenAI-style image / image_url content parts in a chat state and passes them to SGLang; each question is still one prefill.
• How we measured it: the same reconstruction of the Decision Index 0.3.1 vision public suite as for the earlier releases (11 benchmarks, chance-corrected skill and the board's weights), run through the release server, 0 failed requests. We also ran the official vision public suite (12,210 rows, complete).

benchmark	26B-A4B-Gen1 (our reconstruction)	26B-A4B-Gen1 (official vision suite)	stock gemma-4-26B-A4B-it (our reconstruction, same front end)	35B-A3B-Gen1 (our reconstruction)
CV-Bench	72.1	71.9	68.3	75.8
BLINK	58.3	51.7	50.9	59.8
RealWorldQA	51.9	49.8	47.2	71.9
CharXiv	62.2	70.2	49.8	66.9
InfographicVQA	82.0	87.4	73.8	90.7
Mind2Web	71.0	61.2	59.2	86.4
Winoground	78.0	75.8	76.5	80.0
KIE (CORD+FUNSD)	95.4	91.9	92.9	96.2
Moderation (Hateful Memes)	40.8	37.3	42.1	41.1
R-Bench-M	26.2	27.6	29.8	29.1
MMMU-Pro vision	30.9	28.7	25.8	45.1
weighted public index	64.9	63.4	59.7	71.2

• Above the stock model, below the 35B-A3B: 5.2 points higher than the stock gemma-4-26B-A4B-it on our reconstruction (charts, infographics, web screenshots), but 6.3 points lower than KnowLine-35B-A3B-Gen1, mostly on RealWorldQA, web screenshots and exam pages; Hateful Memes and R-Bench-M are slightly below the stock model.
• Indicative only: the reconstruction is ours, not the board's rows; the official-suite column is a self-run of the official kit.

Game harness (KOF '98)

• Setup: single-bout character-mirror matches, 18 games per pair, argmax actions; the same settings as for the other KnowLine releases.
• Result: 15-2-1 against Jev, 17-1-0 against StartLux-Decision-35B-A3B and 8-9-1 against KnowLine-35B-A3B-Gen1; 40-12-2 overall.
• How the games ended: against Jev, 5 of the 15 wins were by K.O. and 10 at time-out; against StartLux-Decision-35B-A3B, 14 of 17 by K.O.; against KnowLine-35B-A3B-Gen1, 5 of 8 wins and 8 of 9 losses by K.O.
• Play style: heavy kick 35%, special_2 32%, heavy punch 16%, special_1 7%, special_3 7%.
• Caveats: 18 games per pair give wide intervals; this is a measured result in this harness, not general fighting-game skill.

Calibration

Computed on the 0.3 public suite using the method described by the Decision Index board: each field is scored right or wrong, confidence is the probability of the chosen option, benchmarks are weighted equally, and 10 equal-width bins are used.

metric	26B-A4B-Gen1 (self-run)	35B-A3B-Gen1 (self-run)	35B-A3B-Gen1 (board)	board median (173 models)
ECE (lower is better)	0.02-0.03	0.04-0.05	0.048	0.078
Brier score (lower is better)	0.27-0.29	0.25-0.27	0.261	0.46
Confidence ≥95% but wrong	1.1-1.2%	1.7%	1.4%
Mean confidence / accuracy	0.82 / 0.78-0.80	0.85 / 0.80	0.85 / 0.80

• Checked against the board: for KnowLine-35B-A3B-Gen1, the board's own figures (ECE 0.048, Brier 0.261) fall inside the ranges our computation gave, so we use the same computation here.
• Temperature in the weights: the final norm weight and the final logit soft-cap include a readout temperature T = 1.25, so every final logit is divided by 1.25 before the softmax over the option labels. The top-ranked option does not change; the probabilities are just less extreme. T was fitted on a set of our own evaluation items whose distribution is covered by the training data, not on the held-out sets or on Decision Index rows. On our held-out sets, ECE with the temperature in the weights is 0.037 (35B-A3B-Gen1: 0.025).
• Why ranges: the board does not publish which 32 benchmarks it uses, so we report ranges over the plausible benchmark sets, as for the earlier releases.
• Recommendation: the temperature was fitted on our data. If your use depends on probability thresholds, check calibration on your own data.

Disclosures

• Decision Index format training data: in the first round, about 34% of the rows target Decision Index benchmarks in their request format or a reworded version of it: train splits of public datasets rewritten in that format (for example WinoGrande XL, ACOS and GSM8K) and synthetic items written in the same format. Another 0.5% are copies of the same items rendered in other formats. The second round is one third injection-invariance items and two thirds a replay of the first round.
• Decontamination: all training data was checked with the same tool as the earlier releases against every Decision Index 0.3 row, all of our evaluation sets and the other public benchmarks we report (Intern-Decision, S1MB, JevBench, MMDM). In the parts built for this release, 491 rows that nearly duplicated an evaluation item, repeated its question or claim, or contained its whole state were removed; the replayed parts come from data that had already passed the same check.
• Selection on evaluations: we compared the first-round checkpoints and the second-round model on our release checks; the release is the second-round model, the only one within our injection limit. It is significantly below KnowLine-35B-A3B-Gen1 on our Chinese sets and Open-Jev 1.1 OOD (see Evaluation).
• No model outputs as labels: no output of Jev or any other decision model was used as a training label.
• Teacher-labelled synthetic data: LLM teachers wrote and labelled our synthetic tasks. The labels were filtered, but not all of them were checked by a human.

Limitations

• Chinese and Open-Jev: clearly lower than KnowLine-35B-A3B-Gen1 on Chinese (C-Eval 74.3 against 87.6; our Chinese in-house sets) and on Open-Jev 1.1. For Chinese-heavy use, prefer the 35B-A3B.
• Size and speed: about 52 GB of bf16 weights, so it needs one GPU with 80 GB or more; each request takes 44.8 ms median, one request at a time on one H20 (see INFERENCE.md).
• Web operation and images: below KnowLine-35B-A3B-Gen1 on our Mind2Web and vision evaluations; there was no image training, so check image use on your own data.
• Prompt injection: planted instructions changed 0.3% of answers on our injection set and 1.8% with unseen templates. Other attacks may still work, so keep untrusted text clearly delimited.
• Private tests: part of the public-suite score comes from benchmarks whose tasks are in our training data, so the 0.3 private tests may show a lower score.

Citation

@misc{knowline26ba4bgen1,
  title  = {KnowLine-26B-A4B-Gen1: a 26B-A4B decision model},
  author = {PelaAI},
  year   = {2026},
  url    = {https://huggingface.co/PelaAI/KnowLine-26B-A4B-Gen1}
}

License

• Weights: Apache-2.0, the same as the base model.
• Code: knowline_server.py is MIT (see the file header).