Dense — completed 5B-token research checkpoint
This is the completed step 9,537, 5,000,134,656 training-target checkpoint, trained from scratch on the frozen FineWeb-Edu corpus. It is an inference release: the original model tensors are preserved exactly, without optimizer or RNG state.
Visibility: public baseline.
Parameters: 341,099,520. Training context: 2,048 tokens.
The 5B endpoint uses the original 15B-token learning-rate schedule, stopped at 5B.
See research_config.json for the actual architecture and training history policy.
Loading and reproducibility
This custom ResearchLM is not a Transformers AutoModelForCausalLM repository.
Follow INFERENCE.md; review the bundled code and use loader.load_release.
Tokenizer files are the frozen Mistral-7B-v0.1 tokenizer used by all models.
release_manifest.json lists file hashes; tensor_manifest.json identifies every
original tensor. provenance.json records the source checkpoint and corpus digests.
The code is derived from the pinned research revision.
Intended use and limitations
Controlled research comparisons, with matched inference policies and total state accounting. This is a small base language model, not instruction tuned. One training seed does not measure training-run variance. No new evaluation or long-context capability is claimed by this publication. Suffix windows apply from the first prompt token, including the current token; dense outer layers still retain history. M4 replaces those ends with recurrent layers; pure Mamba-2 has native recurrence and no suffix-window cap. P32/P64 denote older protected-prefix models, not S-min32/S-min64. These inference choices must not be conflated.
The release does not contain training examples, optimizer state, credentials or
private artifact URLs. Upstream component attribution is in THIRD_PARTY.md.
No new blanket license is granted here beyond the applicable upstream licenses.