OpenSI-0.3B-Base

An open base language model, pretrained from scratch on 2.6B tokens.

Qwen2 architecture · 113.9M params · 2048 context · MIT License

⚠️ The model is 113.9M parameters, not 0.3B

The repo name says 0.3B, but the actual parameter count is 113,932,032 (113.9M). Treat it as a ~0.11B model. See Size reconciliation below.

---

The repo name says 0.3B, but the actual parameter count is 113,932,032 (113.9M). The name is kept for continuity with the project's internal plan (§5.3), not because it describes this checkpoint. Treat it as a ~0.11B model.

Parameters	113,932,032 (113.9M)
Layers / hidden	12 / 768
Heads (GQA)	12 query, 4 KV
FFN (SwiGLU)	2048
Vocab	50,000 (ByteLevel BPE, 2.39 chars/token)
Context	2048
Embeddings	tied
Trained on	2,601,279,471 tokens (11,182,671 documents)

Breakdown: MLP 56.6M (49.7%) · embeddings 38.4M (33.7%) · attention 18.9M (16.6%).

Evaluation results

All numbers are full-set, produced by scripts/bench.py (see Evaluation method below). acc is the primary metric.

Benchmark	n	acc	acc_norm	Random baseline	Verdict
LAMBADA	5,153	46.51% ¹	18.42% ²	~0%	✅ far above random
ARC-Easy	570	32.63%	29.12%	25–27.8%	✅ +5–8 pp
HellaSwag	10,042	26.27%	25.88%	25.0%	❌ +1.3 pp only
C-Eval	1,346	21.62%	27.56%	25.0%	⚠️ acc below random

¹ LAMBADA is single-candidate, so multi-choice acc is meaningless (always 100%). Reported here as greedy token hit rate — whether the target word is the argmax continuation — which is the metric lm-eval-harness uses for this task. ² Whole-word all-correct rate.

Plus, from the deterministic dev pass (13,064,192 tokens):

dev loss / ppl	3.3336 / 28.04 (σ≈0.006)
EOS boundary signal	P(EOS|doc end)=0.611 vs P(EOS|body)=0.000014 → 562×
Overfitting check	sampling σ=0.0930 → no overfitting

What this model can and cannot do

Can: English language modeling. LAMBADA 46.51% against a ~0% random baseline is a large, unambiguous signal that 2.6B tokens of pretraining produced a working language model. Sentences are fluent and well-formed.

Cannot: world knowledge and commonsense reasoning. HellaSwag sits 1.3 pp above random despite a median top-2 margin of 14.6 nats — the model is confidently wrong, not undecided. That task requires understanding what people are physically doing in a scene; fluency does not transfer to it.

Chinese: no academic knowledge. C-Eval accuracy (21.62%) falls below random, and the prediction distribution collapses onto option A (42.3% predicted vs 23.0% actual gold), i.e. a positional bias rather than knowledge. Length-normalized accuracy (27.56%) is barely above random.

The LAMBADA ↔ HellaSwag gap (46.51% vs 26.27%) is the clearest single statement of what this checkpoint is: it learned language, not knowledge.

Why HellaSwag did not meet the original bar

The original acceptance criterion for this tier was HellaSwag ≥35%. Measured: 26.27%. Before accepting that, the following were ruled out one by one: scorer self-checks passed; bf16 noise is ~29× smaller than the median decision margin; trivial baselines (always-first, always-longest, always-shortest) all score 24–25%; the same code scores 32.63% on ARC-Easy, above its own random baseline; and a tokenization-boundary defect was found, quantified (+1.67 pp), and shown to be far too small to close an 8.7 pp gap.

The cause is data, and it is measurable. Training-data language mix by token, not by document:

	docs	tokens (measured)
English	70%	49.9%
Chinese	30%	45.9%

Chinese documents yield ~2.1× more tokens each, so a 70/30 document split becomes a 50/46 token split. HellaSwag is pure English, so this model effectively saw ~1.30B English tokens — and a chunk of that was TinyStories (synthetic simple English), which is close to useless for commonsense about real-world activity.

No scale-up was attempted. Per the project's decision rule, a stage that misses its gate may only be made smaller or shorter, never larger.

Intended use

Research and experimentation on small-scale base-model pretraining. Suitable as a fine-tuning base for English tasks. Not suitable for factual QA, commonsense reasoning, Chinese academic work, or any production deployment.

Evaluation method

lm-eval-harness could not be installed (no PyPI access from this environment), so the four loglikelihood benchmarks are implemented in scripts/bench.py, aligned with lm-eval's doc_to_text / doc_to_target / preprocess. Scorer correctness is enforced by three self-checks that must pass before any number is reported:

• A batched scoring vs a batch=1 reference implementation, per-item logprob
• B index check — the true continuation must score above scrambled continuations
• C padding invariance — identical results alone vs inside a mixed-length batch

Tolerances are dtype-aware: fp32 1e-3, bf16 0.5. (bf16's noise floor was measured at 7.1e-2 against fp32's 5.7e-6, matching the predicted bound for logit magnitude 10.9.)

Export to this Qwen2 format was verified numerically against the training implementation: max absolute error 4.244e-05, cosine similarity 1.0000000000, argmax agreement 100.00%.

One caveat: lm-eval tokenizes context and continuation separately, which for ByteLevel BPE without a leading space produces a boundary that does not exist in the original text ('…on a roof. Heis using…'). This matches lm-eval's behaviour and is kept for comparability; correcting it raises HellaSwag 300-item accuracy 32.00% → 33.67%.

Training details

Optimizer AdamW (β 0.9/0.95, wd 0.1, eps 1e-8), LR 3e-4 → 3e-5 cosine, 2% warmup, grad clip 1.0, global batch ≈ 524k tokens/step (32 × 2048 × 8 accumulation), bf16 with SDPA, torch.compile, seed 1337. 4,961 steps over 14h42m on a single NVIDIA DGX Spark (GB10); train EMA loss 11.0017 → 3.2274 (−70.7%); peak memory 29.8 GB.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("SIZWL/OpenSI-0.3B-Base")
model = AutoModelForCausalLM.from_pretrained(
    "SIZWL/OpenSI-0.3B-Base", dtype="auto", device_map="auto"
)

ids = tok("The capital of France is", return_tensors="pt").to(model.device)
out = model.generate(**ids, max_new_tokens=20, do_sample=False,
                     pad_token_id=tok.pad_token_id)
print(tok.decode(out[0]))

This is a base model, not instruction-tuned. It continues text rather than answering questions, and may not terminate cleanly on a complete sentence — prefer sampling with a repetition penalty for open-ended generation:

out = model.generate(**ids, max_new_tokens=64, do_sample=True, temperature=0.8, top_k=40,
                     repetition_penalty=1.2, eos_token_id=tok.eos_token_id,
                     pad_token_id=tok.pad_token_id)

Data sources

Source	Role
FineWeb	English web text
Wikipedia (zh)	Chinese encyclopedic
TinyStories	Synthetic simple English
Industry Chinese documents	Chinese domain text

See §4.5 of the project plan for the mix and §4.7 for the verification tooling.

License

MIT.

Citation

@online{opensi03bbase,
  title  = {{OpenSI-0.3B-Base}: A 113.9M Open Base Language Model Pretrained on 2.6B Tokens},
  author = {{OpenSI Project}},
  year   = {2026},
  url    = {https://huggingface.co/SIZWL/OpenSI-0.3B-Base},
  note   = {Exported in Qwen2 format. Actual size 113.9M parameters.}
}