ounce100m-v1

A 106,194,240-parameter decoder-only transformer trained from scratch on 999,817,216 tokens of Cion-lab/ounce100m-mix-v1, in 3814 optimiser steps on 2×Nvidia Tesla T4 inside Kaggle notebook sessions. It is the deliverable of a project whose whole point was to find out what a ~100 M-parameter model reaches from ~1 B tokens of a curated public mix, and to report it honestly, including where that is near chance.

Measured facts

Parameters	106,194,240 (recomputed from model.safetensors's own header, not asserted)
Architecture	22 layers, hidden 576, GQA 9 Q / 3 KV heads, SwiGLU 1536, RMSNorm 1e-05, RoPE θ=10000.0, tied embeddings, vocab 49,152
Sequence length	1024 — which is why ARC's 25-shot prompts are truncated from the left by the eval harness
Tokens consumed	999,817,216 (99.98 % of the 1.0 B target)
Batch	micro 4 × accum 32 × 2 ranks = 262,144 tokens/step
Optimiser	adamw_torch, LR 0.0006, warmup fraction 0.02, trapezoid decay over the final 20.0 % of the schedule
Precision	fp16 autocast + fp32 master weights + GradScaler — asserted at the start of every session, and each session's log line precision: fp16=True bf16=False scaler=GradScaler is the evidence
Throughput	12,150 tok/s in the session that produced this export
Attention	eager
Gradient checkpointing	False
Randomness	model seed 20260919, data seed 20260919
Wall clock	4.59 h in the session that produced this export. The run is split across sessions at checkpoint boundaries, so this is not the total training time
Loss	4.43187255859375 -> 4.264701843261719
Validation PPL	75.98 on Cion-lab/ounce100m-mix-v1's held-out val/ shards (22,934,043 tokens, never trained on)

Data

Cion-lab/ounce100m-mix-v1: 1,109,714,831 training tokens in 139 shards + 22,934,043 held-out validation tokens, cut with HuggingFaceTB/SmolLM2-135M:tokenizer.json (49,152 vocab, Apache-2.0), from 15 public sources at ≥15 per shard. Format and loading code are in that dataset's card.

Contamination. Overlap was measured mechanically against benchmark train/validation/dev material of the eight evaluation tasks (13-token windows, counts only — no benchmark item text was ever read, and test splits stayed untouched until evaluation). audit.json in the dataset repo records overlap_total = 0 over 1,073,047 documents and 21,898 held-out documents, tasks_covered = 8/8. Building it required dropping 5878 documents (17,366,967 tokens) that matched reference material; the exact exclusion masks ship in filter/ so the mix is reproducible.

Evaluate it

pip install "lm-eval==0.4.13"
lm_eval --model hf --model_args pretrained=Cion-lab/ounce100m-v1,dtype=float16,trust_remote_code=False \
  --tasks arc_challenge,arc_easy,hellaswag,mmlu,piqa,truthfulqa_mc1,truthfulqa_mc2,winogrande,gsm8k \
  --batch_size 8 --seed 42 --log_samples

No benchmark scores are published with this model. The eight-task evaluation was dropped on 2026-09-21T14:13Z by the project owner before any task produced a score (the first eval kernel died at 1.77 s on a module-path bug and was not relaunched), so every number on this card comes from training or from the held-out validation set - perplexity 75.98 on 1,953 windows of val/, which the model never trained on. The protocol that was going to be used is frozen and public anyway: task ids, metric names, shot counts, splits and chance levels were all fixed in docs/05-eval-plan.md of Cion-lab/ounce100m-code before any score existed, with the driver in eval/run_benchmarks.py at Cion-lab/ounce100m-code. Treat the command above as a reproduction recipe, not as a claim that these results were measured.

Files

file	sha256	bytes
config.json	be63e8d2dc708f8b…	723
cursor.json	c5ea3ba9293f3278…	205
generation_config.json	1dac20576af54bdd…	219
model.safetensors	669ac130a8853cc2…	424,799,272
run_summary.json	d7789a4e70e296c3…	968
special_tokens_map.json	e786b595b9a23148…	831
tokenizer.json	9ca9acddb6525a19…	2,104,556
tokenizer_config.json	4bb9af56a342753d…	3,658

cursor.json is the data position this checkpoint corresponds to: samples consumed, step, seed, the corpus fingerprint and a hash of the visit-order permutation, so the resume can be checked rather than trusted.

Not in this model

No instruction tuning, no RLHF, no chat template — it is a base model, deliberately. It has ~1 B tokens of training against the ~300 B of the comparable published reference models, so expect scores at or near chance on several tasks; that is the experiment's result, not a defect to be engineered around.