ShivankXD/stasis-1

Stasis 1 (base, step 15,106)

Stasis 1 is a 109.5M-parameter decoder-only language model trained from scratch on FineWeb-Edu as a personal research project, entirely on free compute (2× NVIDIA T4). The goal is to study how far a small model can be pushed with careful, verifiable training on a tight compute budget. Weights, modeling code, training logs, frozen evaluation gates and per-item benchmark rows are all released.

Base model, research use only. This is a raw pretrained checkpoint: not instruction-tuned, chat-tuned or safety-tuned. It does not match GPT-2 small (see results) and is not suited to chat, coding or any production use.

Summary

Checkpoint	optimizer step 15,106 (end of the seventh training run)
Parameters	109,529,856
Training tokens	~1.550B cumulative (185.9M in the latest run)
Hardware	free Kaggle 2× T4 (DDP, fp16 AMP), ~2.5 h of training for the latest run
Held-out result (run 7)	all 5 pre-registered gates passed (−0.57% CE on the fresh primary held-out panel vs step 12,269)
vs GPT-2 small	last matched comparison (step 12,269): behind on 5 of 6 benchmarks, ARC-Easy inconclusive, no parity. Step 15,106 has not been compared with GPT-2 yet

Architecture

Type	Decoder-only Transformer (causal LM)
Parameters	109.5M (109,529,856)
Layers / hidden / heads	12 / 768 / 12 (head dim 64)
MLP	SwiGLU, intermediate size 2048
Normalization	Pre-RMSNorm (eps 1e-5)
Positions	RoPE, theta 1e4
Context length	2048 tokens
Vocabulary	32,000, byte-level BPE (tokenizer.json)
Output head	Tied to the input embedding

Training details

Data. English web text from FineWeb-Edu (HuggingFaceFW/fineweb-edu, sample-100BT, pinned revision), packed into 2048-token windows. Held-out panels are drawn so they are domain-disjoint from training documents and are never trained on.

Lineage. Training proceeds in sequential tranches, each resuming the full optimizer state of the previous endpoint.

Stage	Optimizer steps	Tokens in stage	Cumulative tokens
Runs 1–5	0 → 4,126	~830.5M	~830.5M
Run 6	4,126 → 12,269 (8,143 updates)	533.7M	~1.364B
Run 7 (this checkpoint)	12,269 → 15,106 (2,837 updates)	185.9M	~1.550B

Optimizer and schedule (run 6). AdamW (β1 0.9, β2 0.95, weight decay 0.1, grad clip 1.0); 65,536 target tokens per update; 200-update re-warmup to 1.5e-4, cosine decay to 1.5e-5; fp16 AMP (2 skipped updates out of 8,143).

Optimizer and schedule (run 7). Same optimizer, resumed with its full state from step 12,269; fresh FineWeb-Edu shard not used in earlier runs. 100-update re-warmup to 1e-4, then cosine decay toward 1e-5 over a schedule frozen at 3,150 updates after 30 calibration updates. Throughput was lower than the calibration estimate (median 3.12 s/update vs 2.73 s measured during calibration), so the run hit its wall-clock stop at update 2,837 with the learning rate at 1.23e-5. The decay was about 90% complete, not the full planned anneal. 1 skipped AMP update out of 2,837.

Hardware. Free Kaggle 2× NVIDIA T4, PyTorch DDP (NCCL). Run 6: ~2.76 s/update, 7.13 h of training. Run 7: ~3.12 s/update, 2.48 h of training.

Training loss of runs 6 and 7 (per-update raw values and 100-update moving average). Per-update logs for runs 1–5 are not published, so they are not plotted. The jump at step 12,269 is the run-7 re-warmup on a new data shard. Raw logs: metrics/run6_step12269/train_log.jsonl, metrics/run7_step15106/train_log.jsonl. Run 7 training loss went from 3.358 to 3.313 (mean of first vs last 100 updates); last update 3.292.

Results

Frozen acceptance gates (run 7, current checkpoint)

All gates were written to metrics/FROZEN_GATES.json before run 7 started and were not changed afterwards. Both the step-12,269 parent and the step-15,106 candidate were scored live with the same code. 95% upper bounds come from a paired bootstrap (1,000 draws). The primary panel (SEVENTH_FINAL256) is fresh: 256 whole web domains from a shard never used for training, none seen in any training run.

Gate	Step 12,269 → step 15,106	Upper 95% bound	Result
Primary SEVENTH_FINAL256 CE (≥0.5% gain, UB95 delta < 0)	3.3391 → 3.3201 (−0.57%)	delta −0.0177	PASS
SIXTH_FINAL256 no-regression (gain ≥ 0, UB95 < 0)	3.3491 → 3.3301 (−0.57%)	delta −0.0180	PASS
FINAL126 no-regression (gain ≥ 0, UB95 < 0)	3.3977 → 3.3792 (−0.54%)	delta −0.0170	PASS
English93 retention ratio vs early reference (UB ≤ 1.01)	ratio 0.9416	0.9474	PASS
English599 retention ratio vs step 3,403 (UB ≤ 1.01)	ratio 0.9550	0.9568	PASS
SEVENTH_DEV128 (diagnostic only, not a gate)	3.3005 → 3.2815 (−0.58%)	delta −0.0172	n/a

Decision: all five gates passed, so this is recorded as a research gain (RESEARCH_GAIN_ESTABLISHED_NO_AUTOMATIC_PROMOTION). The gain is small: the primary panel cleared the 0.5% threshold by only about 0.07 percentage points. Results: metrics/eval/EVAL_RESULT.json, per-unit rows in metrics/eval/EVAL_ROWS.json.

Frozen acceptance gates (run 6, history)

These gates were frozen before run 6 (metrics/run6_step12269/FROZEN_GATES.json). Comparisons are against the step-4,126 parent.

Gate	Parent → step 12,269	Result
Primary held-out panel CE (≥0.5% gain, UB95 < 0)	3.503 → 3.349 (−4.40%)	PASS
FINAL126 held-out CE	3.523 → 3.398 (−3.57%)	PASS
English93 retention ratio (UB ≤ 1.01)	0.947	PASS
English599 retention ratio (UB ≤ 1.01)	0.960	PASS

Matched comparison with GPT-2 small (measured at step 12,269)

No matched GPT-2 comparison has been run yet for the current checkpoint (step 15,106). The numbers below are for the previous checkpoint, step 12,269 (tag step-12269), and show no parity.

Stasis 1 step 12,269 (109.5M) vs GPT-2 small (124M, openai-community/gpt2), scored by the same script on one T4 in fp32 with no truncation. Metric definitions were frozen before any scoring (benchmarks/gpt2_matched_step12269/METRIC_DEFINITIONS.json, sha256 579ab8c1…afa24137). 95% CIs from a paired bootstrap (10,000 draws). Delta = Stasis − GPT-2.

Benchmark (unit)	Stasis 1	GPT-2 small	Delta [95% CI]	Verdict
WikiText-2 test, full, BPB ↓ (62 articles)	1.2746	1.0565	+0.218 [+0.190, +0.252] (20.6% worse)	GPT-2 better
WikiText-2 256-row panel, BPB ↓	1.5037	1.3179	+0.186 [+0.163, +0.210] (14.1% worse)	GPT-2 better
LAMBADA (OpenAI), accuracy ↑ (5,153)	19.1%	33.7%	−14.6 pts [−15.8, −13.3]	GPT-2 better
LAMBADA target BPB ↓	1.2692	0.7685	+0.501 [+0.484, +0.518] (65.2% worse)	GPT-2 better
HellaSwag val, acc_norm ↑ (10,042)	28.0%	31.3%	−3.3 pts [−4.1, −2.6]	GPT-2 better
PIQA val, acc_norm ↑ (1,838)	59.4%	62.5%	−3.1 pts [−5.0, −1.3]	GPT-2 better
ARC-Easy test, acc_norm ↑ (2,376)	39.6%	40.5%	−1.0 pts [−2.7, +0.7]	Inconclusive
ARC-Easy test, acc ↑	43.7%	43.9%	−0.2 pts [−2.0, +1.7]	Inconclusive

Error bars on the Stasis bars show the 95% CI of the paired delta, placed relative to GPT-2's score.

Bottom line (step 12,269): Stasis 1 is not at GPT-2-small parity. It is clearly behind on language modeling (WikiText, LAMBADA), slightly behind on HellaSwag and PIQA, and statistically indistinguishable on ARC-Easy.

Trend: step 3,403 → step 12,269

The 256-row WikiText-2 panel uses the same method at every checkpoint (GPT-2 re-scored at 1.3179 BPB, reproducing its earlier value).

Step	Stasis BPB	GPT-2 small BPB	Gap
3,403	1.6061	1.3179	21.9% worse
12,269	1.5037	1.3179	14.1% worse

Contamination notes

A 13-gram overlap check (Stasis token space) of every benchmark against the run-6 training tokens (533,965,302 tokens) found shared 13-grams in 12/62 WikiText articles (0.15% of 13-grams), 2/114 panel rows, 2/5,153 LAMBADA items, 23/10,042 HellaSwag items, 4/1,756 PIQA items and 5/2,366 ARC-Easy items. This is descriptive only: data from other runs (including run 7) and GPT-2's WebText were not checked. Details: benchmarks/gpt2_matched_step12269/CONTAMINATION_13GRAM.json.

How to load

# pip install torch safetensors tokenizers huggingface_hub
# see load_example.py for a complete script
import sys, torch
from huggingface_hub import snapshot_download
from safetensors.torch import load_file
from tokenizers import Tokenizer

path = snapshot_download("ShivankXD/stasis-1")
sys.path.insert(0, path)
from frozen_stasis.config import ModelConfig
from frozen_stasis.model import StasisDecoder

model = StasisDecoder(ModelConfig())   # defaults match config.json
model.load_state_dict(load_file(f"{path}/model.safetensors"), strict=True)
model.eval()
tok = Tokenizer.from_file(f"{path}/tokenizer.json")
ids = torch.tensor([tok.encode("The water cycle").ids])
print(tok.decode(model.generate(ids, max_new_tokens=60, top_k=50)[0].tolist()))

This is a custom architecture, so it does not load through transformers.AutoModel.

Roadmap / planned future work

• Continued pretraining on free compute: further resumable tranches on free 2× T4 GPUs, each judged by evaluation gates frozen before the run.
• Matched GPT-2 re-evaluations after each run: the same frozen-metric comparison against GPT-2 small, with confidence intervals, published whether the gap closes or not.
• Later: instruction/chat fine-tuning, then a public chat demo Space. This is planned and not built yet. The current model is a base checkpoint with no chat or instruction-following ability.

Limitations and intended use

• Intended use: research on small-model pretraining, training dynamics, evaluation methodology and reproducibility.
• Out of scope: chat, instruction following, coding, factual question answering, or any high-stakes or production use.
• Small base model trained on ~1.55B tokens; outputs are often repetitive, incoherent or factually wrong.
• Not instruction-tuned or aligned; may reproduce biases or problematic content from web data.
• English only; evaluated only on the panels and benchmarks listed here.

Files

• model.safetensors: step-15,106 weights, fp32 (converted from training endpoint sha256 eb9176ab…c8f2e8; reload verified bitwise identical, strict load OK). Optimizer/recovery state not included. Earlier weights stay available at tag step-12269.
• config.json, tokenizer.json, frozen_stasis/ (exact training modeling code), load_example.py
• benchmarks/gpt2_matched_step12269/: frozen metric definitions, results, run log, contamination report, per-item rows for both models
• metrics/: latest run (run 7) status, config, schedule, frozen gates, eval results/rows, train_log.jsonl; archived per-run copies in metrics/run6_step12269/ and metrics/run7_step15106/. Private storage paths are replaced with <redacted>, so the public FROZEN_GATES.json files no longer hash to the originals (run 6 94b62c38…, run 7 c9e73d49…); no values were changed.
• assets/: charts and make_charts.py to regenerate them

License

Code and weights: Apache-2.0 (see LICENSE). Training data: FineWeb-Edu, ODC-By v1.0, also subject to CommonCrawl's Terms of Use.

Citation

@misc{prabhudessai2026stasis1,
  author       = {Prabhudessai, Shivank Samir},
  title        = {Stasis 1: a 110M-parameter language model trained from scratch on free compute},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/ShivankXD/stasis-1}},
  note         = {Personal research project; checkpoint step 15,106}
}

FineWeb-Edu:

@misc{lozhkov2024fineweb-edu,
  author    = {Lozhkov, Anton and Ben Allal, Loubna and von Werra, Leandro and Wolf, Thomas},
  title     = {FineWeb-Edu: the Finest Collection of Educational Content},
  year      = 2024,
  url       = {https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu},
  doi       = {10.57967/hf/2497},
  publisher = {Hugging Face}
}