Slayer1B 1.09
Arkadiusz SΕota, Anna Sieczka, Piotr StyΕa / Fabryka AI (SlayerLab)
Continued pretraining from 1.08 is paused after spot preemption at approximately 05:15 UTC on 10 November 2026. Real training began at approximately 04:50 UTC on eight B200 GPUs: about 25 minutes of training within a roughly 67-minute machine allocation. This continuation is not the abandoned from-scratch attempt, and its real history is preserved. No completed 1.09 model, final quality verdict, verified local restore or inference-ready weight release is claimed.
| Evidence at this pause | Step |
|---|---|
| Last received public sample, ingested at 05:12:59 UTC | 670 |
| Last saved checkpoint reported by operators; restore not verified here | 600 |
The public sample is not a checkpoint receipt or a live GPU reading. Resume is awaiting approved capacity; successful recovery and local checkpoint copies need their own evidence. An automatic-restart plan does not prove that resumed training has already occurred.
Objective and starting point
The requested 1.09 continues the existing 1B model with diverse, new, deduplicated data, aiming for more than 100B unique tokens after continuation. Inherited content-unique history and the cumulative PL/EN balance remain unresolved; the reviewed source/file-position planning count below does not certify them. Neither replayed exposure nor the old 95,000-update default proves the target. The earlier fixed 21.37% addition is no longer a requirement. The deadline and approved budget remain constraints, not achieved results.
The starting weights are SlayerLab/Slayer1B-1.08, not random initialization or the abandoned attempt's checkpoint. Operators verified step-zero INIT-FROM, fresh optimizer state, approved c01 bytes, fresh numerical control and successful preflight before the observed updates. The selected public v5 launcher uses a separate output history for this continuation.
The unchanged architecture is specified as 1,073,912,851 parameters, 20 layers, width 2,048, 16 heads, SwiGLU, RMSNorm, RoPE base 500,000, QK normalization and value residuals. The reused v6 tokenizer has vocabulary 32,768 and pin 0640d3bd3674d7a6e59c540d945a2ad275a12e1bcaeb6007c37a90751f9815dd.
Architecture and tokenizer lineage includes SlayerLab/Slayer1B-1.07. The selected 1.08 base was published as an experiment with an ARC-Easy retention-guard FAIL; choosing it for further training does not erase that verdict or establish a successful 1.09.
The training configuration records the selected CPT recipe and its interrupted execution, preserving completed 1.08 and the abandoned first attempt as separate historical evidence.
Selected CPT recipe β paused after real execution
| Setting | Operator-selected recipe |
|---|---|
| Planned new training | 80,000 updates Γ 1,048,576 = 83,886,080,000 positions, sixteen 5,000-update slots; not completed exposure |
| New-source allocation | 20% PL / 80% EN target; not a measured cumulative model-language ratio |
| Optimizer / schedule | Muon peak 0.02 plus AdamW peak 0.0006; 2,000-update warmup; final 20% WSD decay to 0.00006 |
| Checkpoints | Full state every 200 updates; slot-end weight milestones; local workstation copies are a plan, not verified here |
| Code | Launcher v5 SHA256 796c71016c1e167bfd3ca98b78518f91c3502a96eac233dd7186ef6639fcc656; exact command and converter pins in the configuration document |
The lineage-based planning count is at least about 18.86B inherited positions disjoint by source files, with an upper estimate of about 23B. The lower count, 18,863,257,682, was recomputed from deterministic sampler coverage and step logs and independently repeated in review; it does not multiply a source share by an epoch fraction. If all 80,000 updates consume new deduplicated positions once, the corresponding planned source-position total is at least 102,749,337,682, approximately 102.7β107B. Historical content-level near-duplicates across sources have not been deducted; this source-position calculation alone does not certify the requested globally content-unique total or actual completion.
Observed CPT start β not a completed model
The separate CPT Track history preserves real training aggregates. Its 02:27 UTC registration timestamp is not the 04:50 UTC training start reported by operators. Fresh hardware control, initialization from 1.08, world size eight and preflight were independently checked for that start.
| First checked public sample | Value |
|---|---|
| Update | 20 |
| Training loss | 2.486659 |
| Interval throughput | 578,293 positions/s |
| New processed positions | 20,971,520 |
| AdamW learning rate during warmup | 0.000006 |
This is an early sample, not the latest step; use Track for current progress. Interval throughput is not full-window or billed throughput, and training loss on the new mixture is not a quality evaluation or directly comparable with 1.08 loss on different data. No completed slot, global content-unique total or checkpoint copy/restore is established by this sample.
Archived from-scratch attempt β not the CPT recipe
| Setting | Abandoned first-attempt configuration |
|---|---|
| Trainer / optimizer | Existing r24.2; Muon with AdamW; fresh optimizer |
| Batch / context | 256 Γ 4,096 = 1,048,576 positions per update |
| Layout | Eight GPU ranks; microbatch 8 Γ accumulation 4; activation checkpointing off |
| Budget | 95,000 updates; nineteen slots of 5,000 |
| WSD | 2,000 warmup updates; constant LR 0.0006, Muon 0.02; final 20% uses 1βsqrt decay to nonzero LR floor 0.00006 |
| Checkpoints | Every 500 updates and retained slot-end milestones |
The abandoned plan's 95,000 Γ 1,048,576 = 99,614,720,000 positions was not exactly 100B and is not the approved length of the revised continuation. Its nineteen planned 5,000-update slots are historical configuration, not a measured unique-history count or evidence that those updates completed.
Slot-1 GPU execution was observed before that historical interruption. Do not resume the from-scratch state as the requested CPT. The revised continuation has its own output history and Track identity; historical points have not been relabelled as training from 1.08. Each queue extension still requires approved byte ranges, manifest digests and preserved sampler identity within its experiment.
Archived first-attempt evidence β not CPT progress
The public Track run is separate from 1.08. At update 30 it records training loss 10.3959, interval throughput 576,593 positions/s and 31,457,280 processed positions (30 Γ 1,048,576). This is an early interval, not end-to-end billed throughput or a quality evaluation. The first slot targets 5,000 updates; its completion is not claimed.
The last received public sample before the pause is update 450, training loss 4.3910, with 471,859,200 processed positions. Its ingestion timestamp is 22:36:50 UTC. It is not a live GPU reading, a final exposure count or proof of a checkpoint at update 500. The trainer's recoverable step must be checked from the saved state, not inferred from elapsed time or the tracker label.
Fresh numerical control and step-zero INIT-FROM preflight were verified for the CPT start described above. The earlier permission for the abandoned slot was not reused as that evidence. Any later physical-host restart still requires its own numerical control; successful start does not prove a checkpoint restore.
Checkpoint copies are currently directed to a trusted local workstation, not Hugging Face. The retention plan is slot-end weight milestones plus the latest full optimizer checkpoint. That plan does not prove a successful local copy or recovery; publication of documents and code does not imply publication of model weights.
Data and exposure
The first fresh CPT reference, c01, has 5,347,737,900 input positions, about 80% English-source FineWeb-Edu and 20% Polish-source GoLLeM web. Operators reproduced it on the training host and checked TRAIN SHA256 3341080241aeabd1e03c77530a6d735cb9501ecd788fba60ea065a944cd09ee1 before the observed CPT start. A complete 5,000-update slot would process 5,242,880,000 positions, not the entire input capacity. Neither available capacity nor the early updates establish slot completion. Older s01βs03 references remain historical, not fresh CPT inputs; see the data card.
The PL/EN balance will follow the approved fresh-data inventory. The user explicitly allows a different ratio; source buckets still do not establish the language share of actually consumed windows. Missing ancestral data must be reported as a limitation, not silently counted as verified unique exposure.
New material requires licensing, PII filtering, contamination scans, grouped held-outs and deduplication against the base model's full data ancestry. A 1% document holdout is not necessarily 1% of tokens. Historical exposure in the starting weights cannot be undone by filtering new inputs; retrospective benchmark hits must be handled explicitly in evaluation denominators.
Evaluation and acceptance
The expanded evaluation plan includes 1.07 and 1.08 reference measurements before future 1.09 milestones. Protocols, model/data digests and denominators must accompany published aggregates; the revised continuation's final acceptance protocol must be established before its results are interpreted.
New benchmarks remain descriptive, contamination check pending until the retrospective checks are complete. No old threshold, early training-loss decrease or retrospectively chosen checkpoint is presented here as acceptance of the revised continuation.
Comparison is descriptive: no replay-only control is budgeted, and initialization, exposure and schedule differ. No causal dataset benefit or multi-seed significance can be claimed.
Cost, licensing and limitations
The declared 1.09 budget is USD 400, including preparation, restarts and evaluation, not just steady-state training. The additional work needed depends on verified inherited and new unique-data counts; the old 95,000-update plan is not a completed budget calculation. Final expense, sustainable throughput and deadline compliance remain unverified.
The selected 1.08 base declares Apache-2.0; source-text rights and future publication still require separate approval. Privacy filters are heuristic; generated text may be inaccurate, biased, offensive or personal. No reasoning quality, instruction-following ability or professional reliability has been demonstrated for the revised 1.09.