Maggio33/GoLLeM-v5-32M

GoLLeM-v5 32M

32M model of the GoLLeM-v5 scaling study (board entry).

Author: Arkadiusz Słota / Fabryka AI. Part of the GoLLeM-v5 series of small English language models trained from scratch. All checkpoints of the series, including intermediate ones, are archived in SlayerLab/gollem-v5-ckpts.

Model

• 31.6M parameters; 6 layers, d_model 576, 9 heads; context 1024 tokens.
• Architecture: decoder-only Transformer (nanoGPT lineage), learned positional embeddings, tied input/output embeddings.
• Tokenizer: BPE, 12,288 tokens (tokenizer.json, sha256 3733307577230bb4802d2c774d8e2323f7f64e4139c13712736a57daf91bdda1), 3.8605 bytes per token on WikiText-2 test.
• Weights: model.safetensors (its sha256 is shown by the Hub for the LFS file), converted tensor by tensor from run_32m_16b/ckpt.pt (sha256 d60c96f25864a57dc9d085af92c9452c15c11b012db1edb18c3303764107b740) without changing any weight. Only weights are published here: no pickled checkpoint, optimizer state or training logs.

Training

• Recipe: AdamW, lr 6e-4 cosine to 6e-5, batch 64 x 1024 tokens, 244,141 steps = 16B tokens, seed 1337, bf16, one RTX 5090.
• Data: ARC-MIX (9.42B BPE tokens): the base mixture SlayerLab/minimal-en-corpus-5b (~5.4B tokens: FineWeb-Edu, DCLM, StackExchange, open-web-math, FineMath, scientific papers, books/Gutenberg, code, CC-News) plus FineWeb-Edu and OpenStax textbooks, with ARC-relevant science, reasoning and Q&A web content upweighted (gold ~3x, related ~2x). Filtered against benchmark test sets with 13-gram matching at build time.
• Result checkpoint: The published checkpoint ckpt.pt is the end of the run (16B tokens).
• At matched tokens it moved past the ~70 BLiMP ceiling seen in the 16M token scan.
• A 32M run at 18B tokens on the same corpus did not improve (BLiMP 72.38): more epochs over a fixed corpus did not help.

Evaluation (our measurement, not official)

Measured by us on the result checkpoint with the Glint Tiny-ML board protocol (BLiMP: 67 configs, 67,000 pairs, sentences clipped to 256 tokens, raw log-prob preference; ARC-Easy: test split, zero-shot, LL(question + choice) - LL(question); WikiText-2: byte-normalized perplexity). These are not official scores.

Benchmark	Score
ARC-Easy (acc, %)	44.44
BLiMP (acc, %)	73.48
WikiText-2 byte perplexity (lower is better)	2.5386
eff (board formula, includes a size multiplier)	75.41
MultiBLiMP-pl (acc, %, 3,272 pairs)	64.27

• eff = mean of BLiMP, ARC-Easy and normalized WikiText-2, times a size multiplier that is larger for smaller models; compare eff only between models under the same board formula.
• This is an English model. Its Polish MultiBLiMP score is close to the length baseline (always choosing the shorter sentence gives 59.67 %); it is reported for completeness, not as Polish ability.
• Board: https://track.fabryka.ai/models (row "GoLLeM-v5 32M").
• Source of the numbers: liczby_v5.json (sha256 35c9022ee9dbd96d588093c6fc69604d0eaabd18843458671205db41e234dde8), built from the anonymous board read-out and our PL measurement files.

Benchmark overlap disclosure

A scan of the finished training corpus found that 7 of 62 WikiText-2 test articles appear almost completely in it (as web copies) and 30 of 2,376 ARC-Easy test questions have a match (mostly short factual sentences). On the 16M, 32M and 64M Muon v1 models, removing them changed WikiText-2 byte perplexity by less than 0.01 and ARC-Easy by -0.07 to +0.19 points, within noise (details in the root card of the archive repository). BLiMP was covered only by the build-time 13-gram filter.

How to use

The weights are a plain PyTorch state dict saved as safetensors; they are not a transformers AutoModel. The model class is GPT in train_gpt_ref.py in SlayerLab/gollem-v5-ckpts; build it from config.json and load the state dict, then trim logits to 12,288.

Limitations

• Small base model trained from scratch for research: not instruction-tuned, limited factual knowledge and coherence, not for production use.
• English only.
• Single seed; scores come from one checkpoint and our own implementation of the board protocol.

Training data

Base mixture: SlayerLab/minimal-en-corpus-5b (research-mix-5b, 5.0B tokens of its own tokenizer). It is an aggregate corpus: no unified licence is applied and each source keeps its upstream terms. Sources as named in its manifests/mixture.json:

source	tokens (M)	upstream licence
fineweb-edu	1,101	ODC-By 1.0
dclm-baseline	801	CC BY 4.0
starcoderdata	469	The Stack terms (per-file licences, opt-out)
stackexchange	447	CC BY-SA
loc-pd-books	402	CC0 1.0
wikipedia	362	CC BY-SA 3.0 / GFDL
open-web-math	228	ODC-By
finemath	221	ODC-By
project-gutenberg	210	public domain in the US
scientific-papers	206	unknown
ultrachat	200	MIT; text generated by OpenAI models
wildchat	151	ODC-By; text from conversations with OpenAI models
cc-news	150	unknown (news articles under copyright)
tiny-textbooks	51	Apache 2.0; model-generated text
open-subtitles	1	film subtitles, under copyright

• Upstream repositories are not recorded in the corpus manifest; licences are those of the Hub datasets matching these source names (not confirmed against the build).
• Added on top of the base mixture for this series: FineWeb-Edu (ODC-By 1.0) and OpenStax textbooks (CC BY 4.0).
• Aggregate corpus; each source keeps its upstream terms. Commercial use: review upstream terms, in particular CC-News, scientific papers, StarCoderData and model-generated chat data.

Licence and attribution

• Weights: no single licence is claimed for the weights yet; they were trained on data under the licences listed in the Training data section above, and any use must respect those terms (license id other, name gollem-v5-see-training-data).
• OpenStax textbooks: CC BY 4.0, © Rice University, https://openstax.org; titles and editions in OPENSTAX_ATTRIBUTION.md in this repository.