MariusNocturnum/DWARF-55M-Base

DWARF-v2 55M Base — FA@L3

DWARF-v2 55M Base is a 55.5M-parameter causal language model with a fixed 2,048-token context window. It is an inference-only release of the FA@L3 hybrid: nine DSQG sparse offset-attention blocks and one dense causal SDPA global-mixer block at layer 3.

This is a base pretrained model, not an instruction-tuned chat model.

Architecture contract

• Hidden size: 512
• Layers: 10
• Attention heads: 8 (head dimension 64)
• MLP size: 1,536
• Vocabulary: 50,282 tokens
• Context length: 2,048 tokens
• Global mixer: causal full attention at layer 3 (FA@L3)
• Sparse blocks: causal DSQG offset attention with 96 fixed offsets partitioned across A/B/C block types

The repository ships a portable PyTorch inference implementation. It intentionally favors inspectability and broad compatibility over the specialized training-kernel throughput path.

Loading

DWARF is a custom Transformers architecture. Review and pin the repository revision in production before enabling remote code.

from transformers import AutoModelForCausalLM, AutoTokenizer

repo_id = "MariusNocturnum/DWARF-55M-Base"
tokenizer = AutoTokenizer.from_pretrained(repo_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(repo_id, trust_remote_code=True)

inputs = tokenizer("The future of sparse sequence models is", return_tensors="pt")
output_ids = model.generate(**inputs, do_sample=False, max_new_tokens=40)
print(tokenizer.decode(output_ids[0], skip_special_tokens=False))

The initial release supports unpadded sequences only. Pass one unpadded prompt per batch and keep the total length, including generated tokens, at or below 2,048.

Training provenance

• Pretraining artifact: pinned Dolma 3 Mix 150B-derived raw/EOD-packed data artifact
• Recorded trainable input-token slots: 9,995,118,211
• Tokenizer: OLMo1 Base ChatML tokenizer, 50,282 tokens

The release manifest contains SHA-256 hashes and the exact checkpoint/model contract used for export.

Evaluation

All results below use lm-eval==0.4.12, the staged release loaded through the Hugging Face hf backend with trust_remote_code=True and dtype=bfloat16 for inference (the stored safetensors weights are FP32), batch size 4 per evaluator process, seed 42, and 0-shot prompts. acc_norm is length-normalized multiple-choice accuracy; where it is unavailable, the reported primary metric is acc.

Benchmark	Split	Examples	Primary metric	Score
ARC-Easy	test	2,376	acc_norm	39.14 ± 1.00
ARC-Challenge	test	1,172	acc_norm	22.18 ± 1.21
HellaSwag	validation	10,042	acc_norm	27.40 ± 0.45
WinoGrande	validation	1,267	acc	51.62 ± 1.40
PIQA	validation	1,838	acc_norm	58.76 ± 1.15
OpenBookQA (main)	test	500	acc_norm	25.80 ± 1.96
BoolQ	validation	3,270	acc	60.09 ± 0.86
LAMBADA (OpenAI)	test	5,153	acc	19.37 ± 0.55

LAMBADA word-level perplexity was 217.88 ± 9.94. The release tokenizer encodes 1,568/5,153 (30.43%) LAMBADA targets as more than one token, so the reported accuracy is the harness's full-target exact continuation metric rather than a single-token proxy.

evaluation_summary.json contains exact values, stderr, task versions, and run metadata. OpenBookQA and BoolQ use local transport adapters over the official public Parquet splits because the host's stored Hub credential could not access the stock public dataset endpoints; their prompts, choices, target mapping, split, and metrics exactly match the installed lm-eval task definitions. Prediction-level artifacts and held-out labels remain in a local evaluation-only audit and are intentionally excluded from the model upload.

Intended use and limitations

This model is for research, experimentation, and analysis of the DWARF-v2 architecture. It can generate inaccurate, unsafe, biased, or fabricated text. It has not been instruction tuned or safety aligned. Do not use it for high-impact decisions, sensitive personal data, or safety-critical workflows.