NextTerm-2-5M

(Model Card Co-Written w/ Opus 5.5)

Model Summary

NextTerm-2-5M is a 4.9M parameter causal transformer trained to continue integer sequences. It is the smallest model in the NextTerm-2 series (5M, 50M and 440M), three models trained on the same 10B tokens with identical hyperparameters for controlled comparison. It uses the Qwen3 architecture with a byte-level vocabulary: the 256 byte values plus BOS, EOS and PAD.

The training corpus, N8Programs/oeis-ladder-10b, contains every training-split OEIS sequence once (terms from its b-file, up to 4,096 bytes) plus 9.2B tokens of distinct, synthetically transformed copies of OEIS sequences.

NextTerm-2-5M is the bottom rung of the series. It continues most arithmetic sequences but rarely higher-degree polynomials, yet it beats the roughly 120x larger Qwen3-0.6B on OEIS-Eval-Neo, Ryskina & Knight and M1.

The model reads integer sequences formatted as comma-separated values, with a trailing comma, for example:

1,-2,3,-4,

Training data contained only digits, commas and -, with no spaces and no leading zeros. Other characters are accepted by the byte-level tokenizer but are out of distribution, so format prompts exactly as above.

Training Details

Field	Value
Parameters	4,856,064
Architecture	Qwen3-style causal LM
Layers	6
Hidden size	256
FFN size	768
Attention heads	2 (head dim 128)
KV heads	1
Vocabulary size	259 (256 bytes + BOS/EOS/PAD)
Training tokens	9,999,999,999 (one epoch)
Sequence length cap	4,096 bytes per sequence
Batch mode	Whole-document packing, 98,304 token positions per step (97% filled), no cross-document attention
Steps	105,029
Optimizer	Muon/AdamW hybrid with μP width scaling
LR schedule	Linear warmup over 2% of steps to 2e-3 for Muon, 5e-4 for AdamW, cosine decay to 0.1x
AdamW	β = (0.9, 0.99), weight decay 0.01
Training hardware	2× NVIDIA DGX Spark (GB10), data parallel, bf16
Training time	2 hours 56 minutes
Export dtype	bfloat16

Muon was used for 2D weight matrices and AdamW for embeddings, norm gains and the output head. All three NextTerm-2 models share these settings.

Evaluation Results

Main Benchmarks

Model	OEIS-Eval-Neo	Ryskina & Knight	M1 Competition 111 MAPE
NextTerm-2-440M	36.25%	66.67%	17.1105
NextTerm-2-50M	31.56%	66.67%	18.3615
NextTerm-2-5M	23.38%	42.11%	20.8527
NextTerm-440M	34.43%	52.63%	17.6239
NextTerm-47M	29.49%	70.18%	18.7621
Qwen3-0.6B	18.44%	33.33%	22.7984
Qwen3-1.7B	20.77%	49.12%	22.2411
Qwen3-4B	23.74%	63.16%	19.1731
Qwen3-8B	24.62%	57.89%	18.4027
Qwen3-14B	26.00%	59.65%	17.9837

OEIS-Eval-Neo is a held-out OEIS next-term evaluation (19,034 sequences). M1 Competition 111 reports macro MAPE, where lower is better; the Naive2 forecasting baseline scores 17.7987. Ryskina & Knight (2021) is a 57-sequence next-term benchmark based on psychometrics and puzzles. NextTerm-47M remains ahead there, consistent with its strength on very short prompts.

Polynomial Continuation

The polynomial continuation evaluation samples integer sequences from polynomials of degree 1 through 4 and asks for the next term. Accuracy is exact match across 200 samples for each prompt length k.

Model	Arithmetic	Quadratic	Cubic	Quartic
NextTerm-2-440M	96.15%	94.39%	88.70%	83.74%
NextTerm-2-50M	96.31%	84.11%	63.95%	47.86%
NextTerm-2-5M	78.21%	23.87%	1.89%	0.50%
NextTerm-440M	94.38%	86.39%	75.20%	67.83%
NextTerm-47M	94.15%	81.07%	37.43%	15.17%
Qwen3-0.6B	90.31%	8.72%	0.30%	0.02%
Qwen3-1.7B	93.10%	41.57%	5.36%	0.71%
Qwen3-4B	93.90%	77.26%	28.18%	5.98%
Qwen3-8B	96.10%	80.59%	32.93%	7.95%
Qwen3-14B	95.60%	84.61%	49.16%	14.98%

Usage

MLX

mlx_lm.generate --model N8Programs/NextTerm-2-5M --prompt "1,2,3,"

Hugging Face Transformers

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "N8Programs/NextTerm-2-5M"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    dtype=torch.bfloat16,
    device_map="auto",
)

prompt = "1,2,3,4,5,"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

outputs = model.generate(
    **inputs,
    max_new_tokens=64,
    do_sample=False,
    eos_token_id=[tokenizer.convert_tokens_to_ids(","), tokenizer.eos_token_id],
    pad_token_id=tokenizer.pad_token_id,
)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

For strict next-term evaluation, stop generation on comma or EOS and parse the text before the first comma as the predicted integer. The tokenizer adds BOS automatically.

Reproducibility

This repository contains the evaluation scripts and data used for the results above:

• eval_neo.py for OEIS-Eval-Neo (and the polynomial sets).
• gen_poly_evals.py to regenerate the arithmetic/quadratic/cubic/quartic sets.
• eval_m1.py for M1 Competition 111 MAPE.
• eval_ryskina.py for Ryskina & Knight.
• oeis_val_neo.jsonl, m1_competition_111.jsonl and ryskina_knight_57.jsonl.
• eval_results.txt for the compact result table.

The model was trained with spark-gpt on the N8Programs/oeis-ladder-10b dataset, whose card documents the data recipe in full.

Citation

@misc{nextterm2_2026,
  author       = {Nathan Breslow and Opus 5.5},
  title        = {NextTerm-2: A Scaling Ladder of Pretrained Transformers for Integer Sequence Prediction},
  year         = {2026},
  publisher    = {Hugging Face},
  howpublished = {\url{https://huggingface.co/N8Programs/NextTerm-2-5M}},
  note         = {4.9M parameter model trained on 10B tokens of augmented OEIS data}
}

Attribution

This model was trained on data from the On-Line Encyclopedia of Integer Sequences (OEIS).

• Source: https://oeis.org/
• License: Creative Commons Attribution-ShareAlike 4.0 (CC BY-SA 4.0)
• OEIS End-User License Agreement: https://oeis.org/wiki/The_OEIS_End-User_License_Agreement