Nexus-Erebus-135M

Nexus-Erebus-135M is a 134M-parameter language model trained from scratch with the Isabel Method (Ideoa Labs). It combines a digit-aware tokenizer with a compact Llama-style decoder to reach a balanced profile across commonsense reasoning, science question answering, and integer arithmetic at the 135M scale.

• Developed by: Malios Dark (Ideoa Labs)
• Model type: autoregressive decoder-only transformer (Llama-style)
• Language: English
• Parameters: 134M

Model architecture

• Decoder-only transformer with tied input/output embeddings, grouped-query attention (GQA), and rotary position embeddings (RoPE).
• Hidden size 640, 25 layers, 10 attention heads (5 key/value heads), context length 1024.
• Vocabulary of 32,000, byte-level BPE with digit isolation: every digit is its own token. This single choice is what makes small-scale arithmetic work, because the model reasons over individual digits instead of arbitrary multi-digit chunks.

Training data

Pretraining (from scratch, 2.66B tokens) on a 2025 English web mixture, by token share:

• Ultra-FineWeb, a re-filtered FineWeb successor (0.45)
• DCLM / DataComp-LM baseline web (0.30)
• Nemotron-CC, public sample (0.17)
• Synthetic integer arithmetic (0.05)
• Templated reasoning and word problems (0.03)

Fine-tuning corpus, built only from official training splits (no validation or test item is ever seen, so evaluation is contamination-free by construction):

• ARC-Easy, ARC-Challenge, OpenBookQA, SciQ, QASC, CommonsenseQA, HellaSwag, PIQA, WinoGrande.
• 150k synthetic arithmetic completions.

Training procedure

Pretraining used bf16 with a cosine schedule. Fine-tuning ran for 2 epochs at a low learning rate.

The important detail is the fine-tuning format. Every example is rendered in completion form, that is context + correct continuation (for example 8 + 5 = 13), rather than a letter-answer form such as Question: ... Answer: D. This is deliberate: the benchmarks are scored by the log-probability the model assigns to each candidate continuation, so the thing worth optimizing is how likely the model makes the correct continuation. A model trained to emit a letter learns to pick a letter, not to prefer the correct completion, and lands near chance on the real scoring. Training in completion form aligns the objective with the evaluation.

Evaluation

All benchmarks are run 0-shot and scored by log-likelihood multiple choice: each candidate answer is appended to the context and the highest-scoring continuation is selected.

Benchmark	Metric	Score
HellaSwag	acc_norm	30.61
ARC-Easy	acc_norm	48.61
ARC-Challenge	acc_norm	25.43
PIQA	acc	64.53
ArithMark-2	acc (length-normalized LL)	66.68

Aggregate = ( HellaSwag + (ARC-Easy + ARC-Challenge) / 2 + PIQA + ArithMark-2 ) / 4 = 49.71

ArithMark-2 is the AxiomicLabs/ArithMark-2.0 held-out arithmetic set; it is a synthetic benchmark and is never used for training.

A note on metric choice (for reproducibility)

HellaSwag and ARC are conventionally reported with length-normalized accuracy (acc_norm), not raw accuracy (acc). Under raw accuracy, longer correct answers are penalized simply for having more tokens, which understates these tasks and can even push ARC-Challenge below random chance. We therefore report acc_norm for HellaSwag and ARC, and acc for PIQA, following common practice. Comparing an acc number for one model against an acc_norm number for another produces non-comparable results; all figures above use the conventions above.

The Isabel Method

The Isabel Method is a small-model recipe with five parts:

1. a digit-isolating tokenizer so arithmetic is learnable at small scale,
2. from-scratch pretraining on recent, high-quality web corpora,
3. arithmetic bootstrapping mixed into pretraining,
4. contamination-free fine-tuning on official training splits only, in completion format, and
5. evaluation that matches the scoring the benchmarks actually use.

How to use

from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("MaliosDark/Nexus-Erebus-135M")
model = AutoModelForCausalLM.from_pretrained("MaliosDark/Nexus-Erebus-135M")

prompt = "8 + 5 ="
ids = tok(prompt, return_tensors="pt")
out = model.generate(**ids, max_new_tokens=8)
print(tok.decode(out[0], skip_special_tokens=True))

Intended use

• Research on small language models, digit-aware tokenization for arithmetic, and from-scratch training recipes.
• English text generation and multiple-choice reasoning at the 135M scale.

Limitations

• English only, with a context length of 1024 tokens.
• As a compact research model, outputs should be verified before use in factual or downstream applications.

Creator: Malios Dark (Ideoa Labs).