MathBananaMind-1.1

MathBananaMind-1.1 is a tiny arithmetic specialist language model trained from scratch by BananaMind.

The model has 2,901,984 parameters and reaches 90.28% on ArithMark 2.0 using the benchmark's official raw summed continuation log-likelihood scoring.

It was trained only on synthetic integer arithmetic. No general web, instruction, conversational, or other language-model datasets were used.

This is a base continuation model, not an instruction-tuned chat assistant.

Model Details

Field	Value
Parameters	2,901,984
Architecture	Custom Llama-style decoder-only Transformer
Layers	9
Hidden size	160
Token embedding size	32
Intermediate size	480
Attention heads	5
KV heads	1
Head dimension	32
Attention style	Grouped-query attention
MLP	SwiGLU
Position embeddings	RoPE
RoPE theta	10,000
Normalization	RMSNorm
Vocabulary size	8,192
Context length	1,024
Token projection	Shared input/output token table with learned projections
Weight format	safetensors
Released checkpoint	Step 29,000
Packed tokens at checkpoint	950,272,000
Model type	Base arithmetic causal LM

The narrow 32-dimensional token table keeps compatibility with the 8k tokenizer while leaving most of the parameter budget for the Transformer blocks.

Tokenizer

MathBananaMind-1.1 uses a custom 8k-token byte-level BPE tokenizer with digit-aware tokenization.

Every decimal digit is represented by its own token:

Token	ID
1	9
2	10
3	11
4	12
5	13
6	14
7	15
8	16
9	17
0	18

Special token IDs:

Token	ID
`<	endoftext
`<	pad
`<	unk

Training Data

The model was trained exclusively on synthetic integer equations in this exact ASCII continuation format:

59 + 45 = 104
(16 / 4) + 44 = 48
3 * 9 + 12 / 1 = 39

The corpus covers:

• Addition, subtraction, multiplication, and exact division
• One, two, and three operators
• Standard operator precedence
• Parenthesized two- and three-operator expressions
• Zero operands
• Boundary and high-carry examples
• One- to three-digit operands in the ArithMark-focused release mix

Field	Value
Unique generated examples	644,655
Unique corpus tokens	10,000,014
Full-run packed tokens	1,000,013,824
Full-run supervised answer tokens	246,037,802
Other datasets used	0

The corpus was streamed repeatedly with a 50% one-operator, 30% two-operator, and 20% three-operator training mix.

Training used answer-only supervision. Context and EOS labels were masked, so the objective directly optimized the continuation tokens scored by ArithMark.

Contamination Guard

The training distribution and surface form were deliberately aligned to ArithMark 2.0. To prevent exact benchmark leakage, all 2,500 normalized ArithMark contexts were loaded before generation and hard-excluded.

The generator dropped 135,536 candidate collisions, deduplicated normalized expressions on disk, and asserted zero exact context leakage before writing the final manifest.

Training Setup

Field	Value
Sequence length	256
Micro batch	32
Gradient accumulation	4
Tokens per optimizer step	32,768
Full-run optimizer steps	30,518
Released checkpoint step	29,000
Optimizer	AdamW
Betas	0.9, 0.95
Peak learning rate	1e-3
Warmup steps	100
LR schedule	Linear warmup with cosine decay
Minimum LR ratio	0.3
Weight decay	0.1
Gradient clipping	1.0
Seed	42

Evaluation

Self-reported evaluation on all 2,500 examples from AxiomicLabs/ArithMark-2.0.

The published score uses raw summed log-likelihood over each candidate continuation, matching the benchmark implementation. The release checkpoint was evaluated in float32.

Benchmark Summary

Model	Parameters	ArithMark 2.0
MathBananaMind-1.1	2.90M	90.28%

Score by Operator Count

Operator count	Correct	Total	Accuracy
1	1,139	1,250	91.12%
2	698	750	93.07%
3	420	500	84.00%

Single-Operator Scores

Operator	Topic	Correct	Total	Accuracy
+	Addition	523	538	97.21%
-	Subtraction	411	438	93.84%
*	Multiplication	105	144	72.92%
/	Division	100	130	76.92%

Accuracy on Every Example Containing Each Operator

These groups overlap because a mixed expression can contain multiple operators.

Operator	Correct	Total	Accuracy
+	1,393	1,514	92.01%
-	1,100	1,208	91.06%
*	745	902	82.59%
/	438	487	89.94%

Score by Topic

Topic	Accuracy
Addition	97.21%
Subtraction	93.84%
Multiplication	72.92%
Division	76.92%
Mixed two operators	95.19%
Parentheses with two operators	90.70%
Mixed three operators	88.84%
Parentheses with three operators	79.46%

Repository Files

File	Description
config.json	Transformers configuration for mathbananamind
model.safetensors	Step-29,000 model weights
tokenizer.json	Custom 8k digit-aware tokenizer
tokenizer_config.json	Tokenizer metadata
generation_config.json	Deterministic generation defaults
configuration_mathbananamind.py	Custom Transformers configuration class
modeling_mathbananamind.py	Custom Transformers causal LM implementation
evaluation_results.json	Machine-readable benchmark summary

Usage

This model uses custom architecture code, so load it with trust_remote_code=True.

Install dependencies:

pip install -U transformers safetensors torch

Run a direct arithmetic continuation:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "BananaMind/MathBananaMind-1.1"

tokenizer = AutoTokenizer.from_pretrained(
    model_id,
    trust_remote_code=True,
)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    trust_remote_code=True,
    torch_dtype=torch.float32,
).eval()

device = "cuda" if torch.cuda.is_available() else "cpu"
model = model.to(device)

prompt = "(16 / 4) + 44 ="
inputs = tokenizer(prompt, return_tensors="pt").to(device)

with torch.no_grad():
    output = model.generate(
        **inputs,
        max_new_tokens=3,
        do_sample=False,
        use_cache=False,
        pad_token_id=tokenizer.pad_token_id,
        eos_token_id=tokenizer.eos_token_id,
    )

continuation = tokenizer.decode(
    output[0, inputs.input_ids.shape[1]:],
    skip_special_tokens=True,
)
print(prompt + continuation)

The model was trained for direct continuations such as expression = answer. It was not trained to follow chat instructions or produce chain-of-thought explanations.

Intended Use

MathBananaMind-1.1 is intended for small-model arithmetic research, continuation-likelihood evaluation, digit-tokenizer experiments, and lightweight local inference.

License

Apache 2.0

Citation

@misc{mathbananamind11,
  title = {MathBananaMind-1.1},
  author = {BananaMind},
  year = {2026},
  publisher = {Hugging Face},
  howpublished = {\url{https://huggingface.co/BananaMind/MathBananaMind-1.1}}
}