OpenSI-0.3B-Seed-5Btok

An open base language model, pretrained from scratch on 5.0B tokens.

Qwen2 architecture · 113.9M params · 2048 context · MIT License

⚠️ The model is 113.9M parameters, not 0.3B

The name inherits from the project's internal plan (§5.3), not from this checkpoint. Actual parameter count: 113,916,672. Treat it as a ~0.11B model.

⚠️ Known defect in 17.6% of the training data

561,628,432 of the 3.18B training tokens contain no document separators at all. 59 of the 116 shards — 57 of the 59 English Wikipedia shards, plus both annotation shards — have zero EOS (id 2). Articles are concatenated end to end with nothing between them, so the model was trained on boundaries that read as continuations.

Found 2026-10-08 while packaging the data for release, after training finished. Not fixed — see Data sources below.

---

Parameters	113,916,672 (113.9M)
Layers / hidden	12 / 768
Heads (GQA)	12 query, 4 KV
FFN (SwiGLU)	2048
Vocab	50,000 (ByteLevel BPE)
Context	2048
Embeddings	tied
Trained on	4,999,610,368 tokens (9,536 steps)
Wall clock	25h30m, single NVIDIA GB10
Train EMA loss	10.9951 → 3.0499 (−72.3%)
Peak memory	29.8 GB

Breakdown: MLP 49.7% · embeddings 33.7% · attention 16.6%.

Parameter-count correction. An earlier checkpoint's model card stated 113,932,032. That figure is wrong by 15,360. The value above was derived layer by layer from the architecture (embedding 38,400,000 + 12 × 6,292,992 + final LayerNorm 768) and matches calib.json exactly.

Evaluation results

All numbers are full-set, produced by scripts/bench.py, all three scorer self-checks passing. acc is the primary metric.

Benchmark	n	acc	acc_norm	Random baseline	Verdict
LAMBADA	5,153	48.43% ¹	20.28% ²	~0%	✅ far above random
ARC-Easy	570	32.28%	26.67%	25–27.8%	✅ +4.5 pp
HellaSwag	10,042	26.23%	26.17%	25.0%	❌ +1.2 pp only
C-Eval	1,346	23.77%	27.71%	25.0%	⚠️ acc below random

¹ Single-candidate task, so multi-choice acc is meaningless (always 100%). Reported as greedy token hit rate — the metric lm-eval-harness uses for LAMBADA. ² Whole-word all-correct rate.

From the deterministic dev pass:

dev loss / ppl	3.2842 / 26.7 (final); last-10 mean 3.1817 ± 0.109, ppl 24.2
Best dev ppl	19.6 @ step 8250
Overfitting check	dev fell monotonically across thirds (61.6 → 28.3 → 25.6) → no overfitting

Doubled training tokens did not move HellaSwag

Compared with the predecessor checkpoint (OpenSI-0.3B-Base, 2.6B tokens / 4,961 steps):

	2.6B tokens	5.0B tokens	Δ
final loss	3.2274	3.0499	−0.18 ✅
LAMBADA	46.51%	48.43%	+1.92 pp ✅
C-Eval	21.62%	23.77%	+2.15 pp ✅
HellaSwag	26.27%	26.23%	−0.04 pp ❌
ARC-Easy	32.63%	32.28%	−0.35 pp ↓

HellaSwag is unchanged after doubling the data. This rules out under-training as the cause.

It also puts pressure on the next most obvious explanation, language mix. This run's unique training data is ~59.1% English / ~37.5% Chinese by token. The predecessor's 53-shard corpus was 49.9% / ~45.9%; this run adds 581.6M tokens of English Wikipedia. Across 1.57 epochs this checkpoint therefore saw **2.95B English tokens** — more than the predecessor in both absolute and relative terms — and HellaSwag still did not move.

What was added is encyclopedic English, and a large share of the remainder is TinyStories (synthetic simple English). HellaSwag tests narrative commonsense plausibility — what a person would plausibly do next — which neither supplies. The evidence therefore points at content type rather than language share: this corpus lacks the everyday narrative text the benchmark draws on. That is a reading of the data, not a tested hypothesis; no ablation was run to separate the two.

This matches the project's own decision rule: a stage that misses its gate may only be made smaller or shorter, never larger. Scaling up was tried and it did not help.

Acceptance bar: not met. The plan's target for this tier is HellaSwag ≥48% (earlier internal bar: ≥35%). Measured: 26.23%.

What this model can and cannot do

Can: English and Chinese language modeling. LAMBADA 48.43% against a ~0% random baseline, together with dev ppl 24.2, show that 5.0B tokens of pretraining produced a working language model. Sentences are fluent and well-formed.

Cannot: commonsense reasoning. HellaSwag sits 1.2 pp above random with a median top-2 margin of 13.9 nats — the model is confidently wrong, not undecided. Fluency does not transfer to reasoning about what people are physically doing in a scene.

Chinese: no academic knowledge. C-Eval acc (23.77%) is below random; the length-normalized variant (27.71%) is barely above it. The gap between C-Eval acc and acc_norm indicates a residual option-position bias.

The LAMBADA ↔ HellaSwag gap (48.43% vs 26.23%) is the clearest single statement of what this checkpoint is: it learned language, not knowledge.

Intended use

Research and experimentation on small-scale base-model pretraining — particularly as a data point for the question "does training longer fix a benchmark deficit?" (this checkpoint's answer: no). Suitable as a fine-tuning base for English tasks. Not suitable for factual QA, commonsense reasoning, Chinese academic work, or any production deployment.

Evaluation method

lm-eval-harness could not be installed (no PyPI access from this environment), so the four loglikelihood benchmarks are implemented in scripts/bench.py, aligned with lm-eval's doc_to_text / doc_to_target / preprocess. Scorer correctness is enforced by three self-checks that must pass before any number is reported:

• A batched scoring vs a batch=1 reference implementation, per-item logprob
• B index check — the true continuation must score above scrambled continuations
• C padding invariance — identical results alone vs inside a mixed-length batch

Tolerances are dtype-aware: fp32 1e-3, bf16 0.5.

Export to this Qwen2 format was verified numerically against the training implementation: max absolute error 4.369e-05, cosine similarity 1.0000000000, argmax agreement 100.00%.

One caveat: lm-eval tokenizes context and continuation separately, which for ByteLevel BPE without a leading space produces a boundary that does not exist in the original text. This matches lm-eval's behaviour and is kept for comparability.

Training details

AdamW (β 0.9/0.95, wd 0.1, eps 1e-8), LR 3e-4 → 3e-5 cosine, 2% warmup, grad clip 1.0, global batch ≈ 524k tokens/step (32 × 2048 × 8 accumulation), bf16 with SDPA, torch.compile, seed 1337. 9,536 steps over 25h30m on a single NVIDIA GB10, peak memory 29.8 GB.

Data was read from 115 train shards (3,183,155,100 unique tokens) plus 1 dev shard (13,065,484 tokens), totalling 4,999,610,368 tokens seen ≈ 1.57 epochs. Composition and known defects are documented under Data sources.

Not implemented in this run: chunked cross-entropy, gradient checkpointing, and RNG state in checkpoints (project plan §4.2–§4.4). All three are unnecessary at this scale — peak memory was 29.8 GB of 121 GB — but they are prerequisites for any larger configuration. Because RNG state is not checkpointed, resuming after a crash does not reproduce the exact trajectory.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("OpenSI-SI/OpenSI-0.3B-Seed-5Btok")
model = AutoModelForCausalLM.from_pretrained(
    "OpenSI-SI/OpenSI-0.3B-Seed-5Btok", dtype="auto", device_map="auto"
)

ids = tok("The capital of France is", return_tensors="pt").to(model.device)
out = model.generate(**ids, max_new_tokens=20, do_sample=False,
                     pad_token_id=tok.pad_token_id)
print(tok.decode(out[0]))

This is a base model, not instruction-tuned. It continues text rather than answering questions, and may not terminate cleanly on a complete sentence — prefer sampling with a repetition penalty for open-ended generation:

out = model.generate(**ids, max_new_tokens=64, do_sample=True, temperature=0.8, top_k=40,
                     repetition_penalty=1.2, eos_token_id=tok.eos_token_id,
                     pad_token_id=tok.pad_token_id)

Data sources

Per-shard token counts, sha256 and decompression instructions are in the accompanying dataset card: OpenSI-SI/OpenSI-0.3B-Seed-5Btok-data.

Source	Shards	Train tokens	Role
opensi-train-*	53	2,601,279,471 (81.7%)	FineWeb · Wikipedia (zh) · TinyStories · industry Chinese documents
shard_*	59	581,616,628 (18.3%)	English Wikipedia, wikimedia/wikipedia 20231101.en — no document boundaries
*_annotations	2	5,949 (<0.01%)	NER/entity labels generated on Wikipedia text — no document boundaries
minix-train	1	253,052 (<0.01%)	MINIX 3 textbook CD source code — see disclosure below
Total (train)	115	3,183,155,100	1.57 epochs over 4,999,610,368 tokens seen
dev	1	13,065,484	held out, same provenance as opensi-train-*

Token-level language mix for this run: ~59.1% English / ~37.5% Chinese / ~3.4% other. The English and Chinese shares of the opensi-train group are inherited from the predecessor corpus's measured 49.9% / 45.9% and were not re-measured for this run; the 581.6M English Wikipedia tokens are added on top of that.

Training-data defect: 17.6% of tokens have no document boundaries

561,628,432 tokens (17.6% of training) carry no EOS (id 2) at all. 59 shards are affected: 57 of the 59 English Wikipedia shards, plus both annotation shards. Inside them the Wikipedia articles are concatenated end to end with no separator between them.

During training this means the model was shown sequences where one article's last token is immediately followed by the first token of an unrelated article, with no stop signal anywhere in that source. The boundary is taught as a continuation.

Evidence	whole-file id==2 count = 0; shard_00000.bin's first 9,592 tokens match the source jsonl line 1 tokenization bit for bit
Cause	the main corpus path (prep_shards.py) inserts EOS between documents and its source comment records this exact trap. The Wikipedia shards were added on 2026-10-05 through a separate path that did not reuse that logic, reintroducing a bug that had already been fixed once
Found	2026-10-08, while packaging the corpus for release — after training had finished
Status	not fixed. The published dataset is byte-for-byte what this checkpoint was trained on. Fixing it would break correspondence with the weights and defeat the purpose of releasing the data; a corrected corpus should be a separate release

What it does not explain. It would be convenient to attribute the HellaSwag result to this defect. That is not supported: the two facts coexist and no experiment has been run to separate them. Nothing here should be read as blaming the benchmark score on the missing boundaries.

Training-data disclosure: MINIX 3 source code (0.0051%)

253,052 tokens (0.0051% of the total) come from the source code on the MINIX 3 textbook companion CD (Tanenbaum, Operating Systems: Design and Implementation, 3rd ed.). Its license status is UNVERIFIED: the disc carries no LICENSE/COPYING file, none of the 91 .c files have license headers, and readme.txt states CD-ROM Copyright 2006 Pearson Education, Inc. ... All rights reserved.

This material was included by explicit, recorded decision, acknowledging that it does not satisfy the project's own permissive-license screening rule. It is disclosed here rather than omitted so downstream users can make an informed judgement. The corresponding held-out evaluation split is distributed separately as a private dataset for the same reason.

License

Model weights: MIT.

Training data is released separately under CC BY-SA 4.0 — the corpus contains English and Chinese Wikipedia, whose ShareAlike term governs the collection. Per-source terms (ODC-BY for FineWeb, CC BY-SA for the Wikipedia slices, upstream terms for the rest) are listed in the dataset card.

This differs from the predecessor corpus (53 shards, declared Apache-2.0), which was never published publicly — it exists only in the project's internal private repo. Its content is the opensi-train-* half of the dataset above. That slice also did not include English Wikipedia.

Citation

@online{opensi03bseed5btok,
  title  = {{OpenSI-0.3B-Seed-5Btok}: A 113.9M Open Base Language Model Pretrained on 5.0B Tokens},
  author = {{OpenSI Project}},
  year   = {2026},
  url    = {https://huggingface.co/OpenSI-SI/OpenSI-0.3B-Seed-5Btok},
  note   = {Exported in Qwen2 format. Actual size 113,916,672 parameters.}
}