Flash-Archer-150M-2.0

A clean, decoder-only transformer language model trained from scratch on FineWeb-Edu — 150M parameters, ~2.1B tokens, on a single Tesla T4 in ~7 hours. Built as a self-contained, reproducible pretraining project: custom byte-level BPE tokenizer, streaming data pipeline, cosine LR schedule with warmup, and a hand-written PyTorch training loop.

This is the v2.0 release: the same architecture and training recipe, packaged as a standard HuggingFace LlamaForCausalLM so it works everywhere out of the box — transformers, vLLM, TGI, llama.cpp, pipelines — with bit-identical inference to the original training/inference scripts (verified: logit max-diff < 1e-5).

Model details

Field	Value
Architecture	Decoder-only transformer (Llama-family: pre-norm RMSNorm, RoPE, SwiGLU, tied embeddings)
Parameters	150.3M total (~138M non-embedding)
Layers	18
Hidden size	768
Attention heads	12 (head dim 64)
MLP hidden dim	2304 (SwiGLU)
Context length	1024 tokens
Vocabulary	16,000 (byte-level BPE, trained on 200k FineWeb-Edu docs)
Special tokens	<pad> 0 · <unk> 1 · <bos> 2 · <eos> 3
RoPE θ	10,000
Training precision	fp16 with gradient scaling
Training tokens	~2.12B (step 16,200)
Training data	HuggingFaceFW/fineweb-edu sample-10BT
Hardware	1× Tesla T4 (15.6 GB VRAM), Kaggle
Training time	~7 hours
Final val loss	2.95 · val perplexity 19.2
Optimizer	AdamW (β₁=0.9, β₂=0.95, wd=0.1, cosine schedule, lr 3e-4 → 3e-5, 2% warmup)
Batch	micro-batch 1 × 1024 tokens × 128 grad-accum = ~131k tokens/step
License	Apache 2.0

Quickstart

from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "Norman89107/Flash-Archer-150M-2.0"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype="auto")
model.eval()

prompt = "Photosynthesis is the process by which"
inputs = tokenizer(prompt, return_tensors="pt")
output = model.generate(
    **inputs,
    max_new_tokens=120,
    temperature=0.8,
    top_k=50,
    top_p=0.95,
    eos_token_id=tokenizer.eos_token_id,
    pad_token_id=tokenizer.pad_token_id,
)
print(tokenizer.decode(output[0], skip_special_tokens=True))

Sampling tips

• Lower temperature (0.3–0.6) → more focused, factual output
• Higher temperature (0.9–1.2) → more creative, more repetition
• top_k=0 disables top-k; top_p=1.0 disables nucleus sampling
• Context is 1024 tokens; longer prompts are cropped from the left

Training recipe

The full training code, tokenizer training, and streaming data pipeline are in the accompanying notebooks (archer-150m.ipynb for training, archer-150m-inference-colab.ipynb for inference). Key design choices:

• Streaming, never materialized — FineWeb-Edu is streamed token-by-token; a fixed 2,000-doc holdout is reserved for validation and never seen by training.
• Packed sequences — documents are concatenated with <eos> separators into fixed 1024-token chunks, so there is zero padding waste.
• Activation checkpointing + micro-batch size 1 + gradient accumulation 128 — keeps the effective batch at ~131k tokens/step while fitting 1024-token training in 16 GB VRAM.
• Persistent checkpoints — every 200 steps the model + optimizer + logs are mirrored to the HuggingFace Hub, so a crashed Kaggle session loses at most ~200 steps.

Intended use & limitations

• Intended use: a compact, fast, English language model for experimentation, education, and as a baseline for small-model research. It writes coherent, on-topic prose and follows simple instructions.
• Limitations: at 150M parameters it will hallucinate facts, repeat itself (especially at high temperature), and struggle with multi-step reasoning, math, and code. It has no safety alignment or RLHF — do not use it for applications without additional safeguards. Trained on web data; it may reflect biases and content present in FineWeb-Edu.

Conversion & reproducibility

This release was converted from the original training checkpoint (ckpt_step0016200.pt) into standard HuggingFace LlamaForCausalLM format. The only architectural difference is the RoPE convention (interleaved → half-split), handled by a fixed Q/K head-dimension permutation during conversion. Inference parity was verified: the converted model's logits match the original model's to within float32 rounding (max |diff| < 1e-5), and seeded generation produces identical output. See convert_to_hf.py and verify_parity.py in the project source.

Citation

If you use this model, please cite the training data and this repository:

@misc{flash-archer-150m-2,
  title  = {Flash-Archer-150M-2.0},
  author = {Norman89107},
  year   = {2026},
  howpublished = {\url{https://huggingface.co/Norman89107/Flash-Archer-150M-2.0}},
  note   = {Decoder-only transformer trained from scratch on FineWeb-Edu (sample-10BT), ~2.1B tokens.}
}

@dataset{fineweb-edu,
  title  = {FineWeb-Edu},
  author = {Hugging Face},
  url    = {https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu}
}