Metallum-550M

Metallum-550M is the historical first-generation ML/LLM-engineering specialist from Home Brewed Labs. The project was developed under the codename MetaLLM-V3 and predates Metallum-1B. It is being released as a reproducibility and archival artifact, not as a replacement for Metallum-1B.

The model is a 550.3M-parameter decoder-only language model pretrained from random initialization on one RTX 5090, followed by retrieval SFT, capability/instruction SFT, and one bounded ReST-EM code round. It is deliberately narrow: use it for ML-engineering experiments, technical continuation, and retrieval research—not as a general chatbot.

Loading

This is a custom architecture. trust_remote_code=True is required because stock Llama implementations omit its QK normalization and interleaved NoPE layers.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "HomeBrewedLabs/metallum-550m"
device = "cuda" if torch.cuda.is_available() else "cpu"
dtype = torch.bfloat16 if device == "cuda" else torch.float32

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    trust_remote_code=True,
    torch_dtype=dtype,
).to(device).eval()

prompt = "Explain why QK normalization can stabilize attention training."
inputs = tokenizer(prompt, return_tensors="pt").to(device)

with torch.no_grad():
    output = model.generate(
        **inputs,
        max_new_tokens=160,
        do_sample=True,
        temperature=0.6,
        top_k=40,
    )

print(tokenizer.decode(output[0], skip_special_tokens=True))

There is no chat template. Supply a plain-text continuation or instruction prompt. The bundled inference shim has no KV cache, so generation recomputes the prefix at every step and is slower than similarly sized models with cached decoding. For production use, pin a repository revision and inspect the bundled Python files before enabling remote code.

Architecture

• 28 decoder layers
• hidden size 1,280
• grouped-query attention: 20 query heads / 10 key-value heads
• SwiGLU, intermediate size 3,456
• RMSNorm pre-normalization and per-head QK normalization
• RoPE with theta 500,000, computed through the model's fidelity-preserving path
• no positional encoding in every fourth layer (interleaved NoPE retrieval layers)
• tied 32,000-token byte-level BPE embeddings
• native context length 2,048 tokens
• bfloat16 weights

The precise parameter count is 550,251,264.

Training

Pretraining ran for 4.0B tokens. Its stable sampling pack contains 2.272B packed tokens: 1.351B code (59.5%), 866.7M ML-arXiv text (38.2%), and 54.4M synthetic technical text (2.4%). The final 10% used an 880.9M-token quality-decay pack containing ML-arXiv text, higher-scored ML code, and synthetic technical text. Pack counts include intentional resampling and are not counts of unique source text.

Post-training added synthetic retrieval examples, teacher-derived instruction examples, and execution-verified in-domain ReST-EM examples with retrieval and capability replay.

Training sources include code obtained through the BigCode the-stack-dedup corpus, arXiv-derived ML papers, and synthetic/teacher-derived material. Source-code and paper licenses vary upstream. The Apache-2.0 license in this repository covers the released weights and code; it does not relicense upstream training data. See NOTICE for the provenance disclosure.

Evaluation

The following results belong to the exact released checkpoint, ckpt_500m_v3_restem_r1/sft_step_000250.pt:

Capability	Evaluation	Result
ML knowledge	250-question cloze MCQ, normalized accuracy	0.416
Answer-letter binding	Same 250 questions, MCF format	0.236
In-domain ML code	Internal 40-task executable suite, pass@1	10/40 (0.250)
Retrieval	RULER-style suite at 1,024 and 2,048 tokens	88/90 (0.978)
Retrieval	Synthetic needle suite	778/800 (0.973)
Passkey subset	Synthetic needle suite	499/500 (0.998)
Key-value subset	Synthetic needle suite	279/300 (0.930)
General code	MBPP, pass@8	0.000

On the same internal cloze harness, the campaign recorded 0.280 for Qwen2.5-1.5B and 0.312 for SmolLM2-1.7B. These comparisons are narrow, in-domain measurements—not general model rankings.

All listed internal suites were used during development and checkpoint selection, so the results are selection-aware. This lineage did not receive the later sealed-holdout protocol used for Metallum-1B. Treat the numbers as reproducibility records rather than untouched final estimates.

Intended use

Suitable uses include:

• studying small domain-specialist language models;
• ML/PyTorch technical continuation and scaffolding;
• experiments with NoPE retrieval layers and QK normalization;
• reproducing the first Metallum generation.

Out of scope: general factual QA, general-purpose coding, autonomous code execution, safety-critical decisions, or deployment as an aligned assistant.

Limitations

• This is not a general chatbot and has no preference or safety alignment.
• Free generation is fluent but can make local factual errors; verify technical claims.
• Knowledge measured through cloze scoring does not transfer reliably to answer-letter output.
• General coding is weak; the measured MBPP result is 0% pass@8.
• The native context window is 2,048 tokens. Retrieval results above do not establish behavior beyond that window.
• The inference shim does not implement a KV cache.
• attention_mask is accepted for API compatibility but padding-aware batched inference was not part of the released evaluation path. Prefer unpadded single-example inference or equal-length batches.

Relationship to Metallum-1B

Metallum-550M is an earlier, independently trained checkpoint. Metallum-1B is the maintained successor with a larger corpus, a separate evaluation record, a sealed final holdout, and a constrained-decoding serving wrapper. The two repositories do not share weights.

License and citation

Weights and repository code are released under Apache-2.0. Training data retains its upstream terms; see NOTICE.

@misc{homebrewedlabs2026metallum550m,
  title        = {Metallum-550M},
  author       = {{Home Brewed Labs}},
  year         = {2026},
  howpublished = {Hugging Face model repository},
  url          = {https://huggingface.co/HomeBrewedLabs/metallum-550m}
}