KnowLine-2B-Gen1

Inference notes · 中文 · weights Apache-2.0 · released together with KnowLine-0.8B-Gen1 · also: KnowLine-9B-Gen1, KnowLine-4B-Gen5.5

KnowLine-2B-Gen1 is a 2B decision model based on Qwen3.5-2B. It maps a structured state and one or more typed questions to probabilities over the supplied options: choice, yes/no and ordered score questions. It reads the probability of each option label directly and does not generate free-form text when it answers. KnowLine is developed by PelaAI, a Japan-based team.

Highlights:

• Decision Index 0.3 public suite: 52.23 (self-run, fresh full run with the files in this repository, read twice; 140,178 of 140,178 scoreable requests answered, no errors). The board's full score comes from the maintainers' private run and is not yet available for this model.
• Small and fast: one request at a time on one H20, 30.0 ms median and 60.6 ms mean.
• Our other evaluations: in-house held-out sets 64.9 / 66.6 / 58.5 (English / Simplified Chinese / Traditional Chinese), C-Eval 67.0, Open-Jev 1.1 test / OOD 80.1 / 77.4, JevBench / JevBench-hard 77.5 / 55.9, decision tasks from new domains (our own, 45 tasks) 32.1; see Evaluation.
• Planted instructions: 2.6% of answers moved to a planted option with injection templates seen in training, and 6.3% (38 of 600) with templates not seen in training, above our 5% target; see Limitations.
• Calibrated probabilities: a readout temperature T = 1.32 is folded into the weights. Computed with the board's method on the public suite, ECE is 0.016-0.025.
• S1MB (English v1, local run): Task Avg 44.85 over 137 benchmarks (26,269 decisions, none failed or abstained).

What it does

• choice questions with named options;
• yes/no questions;
• ordered score questions;
• a readout temperature folded into the language-model norm, so the front end reads at temperature 1;
• read twice for short text-only requests (up to 3,000 tokens read once).

Quickstart

Install PyTorch with CUDA, transformers and SGLang 0.5.21, then start the included front end:

bash serve_knowline.sh /path/to/KnowLine-2B-Gen1 0 8080

The front end serves /v1/systemone:

curl -s http://127.0.0.1:8080/v1/systemone \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "m",
    "state": "The parcel arrived damaged.",
    "questions": {
      "refund": {"type": "noul", "instructions": "Should the customer receive a refund?"},
      "route": {"type": "choice", "instructions": "Which team should handle this case?", "criteria": {"billing": "Payments and refunds", "support": "Technical troubleshooting"}}
    }
  }'

• Front end: knowline_server.py, knowline_engine.py and knowline_layout.py implement the scoring front end used for every result below; temperatures.json must sit next to serve_knowline.sh.
• Thinking off: the front end renders prompts with thinking turned off (enable_thinking=False). The Qwen3.5-2B chat template in this repository turns thinking on by default, so pass enable_thinking=False if you build prompts yourself.
• Full settings: see INFERENCE.md.

How Gen1 was made

• Base model: Qwen/Qwen3.5-2B.
• Training: one epoch of LoRA SFT (rank 32, alpha 64, language-model linear layers only), merged into bf16 weights. The released checkpoint is the end of training.
• Training data: the same decision dataset as KnowLine-9B-Gen1, 1,143,656 rows rendered with the Qwen3.5 chat template, at most 12,288 tokens per row.
• Readout temperature: T = 1.32, fitted on our golden sets and folded into model.language_model.norm.weight (see CALIBRATION.json). temperatures.json divides score-question logits by 0.7576 (= 1/1.32), so score questions are read at an effective temperature of 1.0.
• Images: no image training; the vision tower is the base model's.

The training data is not included in this repository.

Comparison

Board entries of a similar size, from the Decision Index board snapshot multimodalart/jev-decision-index @ c130faf6 (data generated 2026-10-29 14:04 UTC). The board's DI 0.4.1 full score comes from the maintainers' private run.

model	size	DI 0.3 public	DI 0.4.1 full	source
KnowLine-2B-Gen1	2B	52.23	not yet scored	self-run, official kit
StartLux-Decision v1.1 2B	2.3B	59.23	40.11	board
baselabs joey-1 (2B)	2.7B	63.30	35.91	board
Jev-Style 2B v4	2.2B	44.37	34.53	board
Sieve 2B v3	2.3B	53.51	34.45	board

Evaluation

Full public Decision Index 0.3

Measured on 2026-10-29 with the files in this repository (bf16, read twice):

measurement	result
Public Decision Index	52.23
Public raw index	64.24
Requests / scoreable requests	140,620 / 140,178
Scoreable answers / errors / unsupported	140,178 / 0 / 0
One request at a time: median / mean / 95th percentile	30.0 / 60.6 / 214.9 ms

• How the run was made: a fresh run of the 0.3 public suite with the official kit (commit 62d2f51), sent to four identical H20 servers started with serve_knowline.sh, 32 client shards. No request needed a retry and no input was truncated.
• Consistency check: 300 random requests (477 answers) sent one at a time to one server: 476 answers were the same as in the full run; the other is a near-tie (top two options within 0.06).
• Hardware: H20 measurements, not the board's reference hardware.

Held-out and other evaluations

Both models of this release, on the same files and serving:

suite	2B-Gen1	0.8B-Gen1
In-house held-out set, English	64.9	57.9
In-house held-out set, Simplified Chinese	66.6	63.5
In-house held-out set, Traditional Chinese	58.5	53.4
C-Eval (4 categories, macro)	67.0	56.3
Open-Jev 1.1 test / OOD	80.1 / 77.4	71.5 / 69.7
JevBench / JevBench-hard	77.5 / 55.9	71.9 / 49.5
Decision tasks from new domains (our own, 45 tasks)¹	32.1	23.5
Prompt injection, seen templates: answers moved (lower is better)	2.6%	2.1%
Prompt injection, templates not seen in training: attack success (lower is better)	6.3% (38/600)	4.2% (25/600)

• ¹ Our own decision tasks from domains not in training, generated programmatically; some share rule-reading skills with our training data.

S1MB (English v1)

A local run of the S1MB English v1 category (evaluator 9ad06fea, dataset 10bcb8c6) against this repository at commit 9f3cde88, run ID knowline-2b-gen1-en-20261029-001: 137 benchmarks, 18,701 cases, 26,269 decisions, none failed or abstained. Task Avg 44.85 (choice 57.4, yes/no 48.4, score 28.7). This is the evaluator's task average, not a Borda rank or a leaderboard position.

Calibration

Computed on the 0.3 public suite with the method described by the Decision Index board (benchmarks weighted equally, confidence = probability of the chosen option, 10 equal-width bins); ranges cover the plausible benchmark sets, because the board does not publish which ones it uses.

metric	KnowLine-2B-Gen1
ECE (lower is better)	0.016-0.025
Brier score (lower is better)	0.36-0.38
Confidence ≥ 95% but wrong	0.6-0.7%
Mean confidence / accuracy	0.73-0.74 / 0.70-0.72

• On our held-out sets: with score questions read at an effective temperature of 1.0, ECE is 0.045; a temperature refit on the released serving gives 1.10. The folded temperature was fitted with every question type at T = 1.32.
• Recommendation: check calibration on your own data if your use depends on probability thresholds.

Disclosures

• Training data: 1,143,656 rows, the same mix as KnowLine-9B-Gen1: 873,134 rows of our earlier decision data, 241,781 further rows of the same sources, 16,824 rows that teach the model to ignore instructions planted in the state, and 11,917 rows of paired-convention rule tasks. It includes public benchmark training splits and programmatically or teacher-generated decision examples; about 32% of the rows are in the Decision Index request style. The per-channel manifest is in TRAINING_SOURCES.json.
• Decontamination: the complete file was screened with standard and strict decontamination against the Decision Index public suite and our evaluation sets before training. This does not exclude exposure during the base model's pretraining.
• Benchmarks in training: several public benchmarks of the Decision Index have training splits in our data, so part of the public-suite score comes from benchmarks whose tasks were trained; the private tests may show a lower score.

Limitations

• Small model: a 2B model is well below our larger models on knowledge and reasoning-heavy questions (for example C-Eval 67.0 and JevBench-hard 55.9).
• Planted instructions: with injection templates not seen in training, 6.3% (38 of 600) of answers moved to the planted option; our target is at most 5%, which this model does not meet. Put untrusted content in data fields of the state, not in the instructions, and keep it clearly delimited.
• Calibration on our sets: with score questions read at an effective temperature of 1.0, held-out ECE is 0.045.
• Images: no image training; image use is zero-shot.
• Thinking: use the included front end or pass enable_thinking=False; with the chat template's default the model would be prompted in thinking mode, which was not used in training.
• Long requests: limited by the model and serving context; the front end does not truncate a request.
• Not a chat assistant: it is a structured decision readout model.

Files and license

The weights and code are released under Apache 2.0. SHA256SUMS covers every file in this repository. The Qwen3.5-2B base model remains subject to its own license and terms.

Citation

@misc{knowline2bgen1,
  title        = {KnowLine-2B-Gen1},
  author       = {PelaAI},
  year         = {2026},
  howpublished = {Hugging Face model repository}
}