MathScout v0.1: Compact Symbolic Arithmetic Transformer (6.31M)

MathScout is an experimental 6.31-million parameter autoregressive micro-transformer developed as an exploratory research and study project. It serves as an empirical Proof of Concept (PoC) investigating whether small-scale language models can learn strict, deterministic symbolic execution through a character-level computational scratchpad (Chain-of-Thought), rather than relying on statistical pattern matching.

Research Status: This model is strictly an experimental prototype and an academic research artifact. It does not have the level of maturity, robustness, or comprehensive validation required for operational, commercial, or production use.

1. Research Motivation and Study Goals

The Problem Under Study: Arithmetic in Neural Models

Standard generative language models, even those scaling to billions of parameters, frequently struggle with multi-digit calculations, fractions, and nested expressions. Because standard language models predict text via statistical distribution over tokens without an intrinsic execution stack, they frequently produce high-confidence hallucinations on elementary arithmetic.

Research Hypothesis and Scope

This study explores the following research questions:

Algorithmic Scratchpad Feasibility: Can a micro-transformer (under 10M parameters) reliably execute multi-step algorithms (column-by-column addition with carries, subtraction with borrows, long division with remainders, fraction reductions via GCD, and operator precedence) when forced to generate explicit intermediate tokens?
Minimal Working Capacity: What is the architectural capacity required for an autoregressive network to maintain state across long scratchpad traces without losing track of positional carries or parentheses?
Curriculum Learning Dynamics: How do progressive training stages combined with memory replay buffers affect catastrophic forgetting in small symbolic models?

2. Model Architecture

MathScout implements a standard decoder-only transformer architecture with rotary positional encodings and no positional bias.

Architectural Parameter	Specification	Purpose / Notes
Total Parameters	6,307,840 (6.31M)	Small-scale experimental research footprint
Transformer Layers (n_layer)	8	Depth required to route attention across scratchpad steps
Attention Heads (n_head)	8	Multi-head attention across operand positions
Embedding Dimension (n_embd)	256	Compact internal latent dimension
Head Dimension (head_dim)	32	Dimension per attention head
Intermediate MLP Dimension	1024 (4x expansion)	Non-linear feature transformation with GELU activation
Positional Encoding	Rotary Position Embedding (RoPE)	Base 10000.0; relative positional awareness without static tables
Normalization	LayerNorm without bias	bias=False, epsilon=1e-5 for numerical stability
Weight Tying	Yes	Token embedding (wte) is directly tied to the output head (lm_head)
Vocabulary Size	64 tokens	Character-level ASCII digits, operators, and 6 control tokens
Context Window (block_size)	512 tokens	Working buffer for intermediate computational steps

3. Mathematical Operations Evaluated in the Study

Within the experimental evaluation benchmark, the model was tested across several mathematical domains:

Multi-Digit Addition and Subtraction:
- Numbers from 1 to 5 digits with carry tracking and borrow mechanics.
- Signed results (for example, 12 - 45 = -33).
Multi-Digit Multiplication:
- Single and multi-digit factor multiplications decomposed by columns.
Long Division:
- Integer divisions producing explicit quotients and remainders (for example, 125 / 6 yields q=20 r=5).
Fractions:
- Common and distinct denominator addition and subtraction.
- Fraction multiplication and division.
- Reduction to irreducible fractions via GCD (using the simplify keyword).
Decimals:
- Decimal addition and subtraction with alignment steps.
Percentages:
- Percentage calculation (for example, 25 % of 80) and conversions (20 % as frac, 15 % as dec).
Operator Hierarchy (PEMDAS):
- Multi-operator expressions with precedence and nested parentheses.
Auxiliary Primitives:
- Magnitude comparison (cmp), digit counting (len), and integer successor (succ).

4. Input and Output Protocol

Prompt Syntax

Expressions must start with the control token <|op|> :

<|op|> {expression}

Formatting Guidelines

*Binary Operators (+, -, , /): Must have single spaces around them (for example: 15 + 28, 25 * 14, 84 / 4).
Fractions (/): No spaces between numerator and denominator (for example: 3/4 + 2/5).
Decimals (.): The decimal point must be directly attached to digits (for example: 3.25 + 1.4).
Parentheses: Spaces around outer operators, no spaces inside brackets: 2 * (3 + 4 * 5).
Percentages (%): Surrounded by spaces: 25 % of 80.

Example Trace

Prompt:
<|op|> 2 * (3 + 4 * 5) 

Model Generation:
<|think|> [step1] 4*5=20 [sub] 2*(3+20) [step2] 3+20=23 [sub] 2*23 [eval] 2*23=46 [res] 46 <|end|>

5. Experimental Model Artifacts and Verification

For reproducibility and research verification, the model weights have been exported into common open-standard formats:

model.onnx (24.2 MB): ONNX graph for runtime testing.
vocab.json (64 tokens): Token mapping dictionary.

Running with ONNX Runtime (Python)

import json
import numpy as np
import onnxruntime as ort

session = ort.InferenceSession("export/model.onnx")
with open("export/vocab.json", "r", encoding="utf-8") as f:
    stoi = json.load(f)
itos = {v: k for k, v in stoi.items()}

def run_mathscout(expression):
    prompt = "<|op|> " + expression 
    tokens = [stoi[c] for c in prompt]
    end_id = stoi["<|end|>"]

    for _ in range(512):
        input_ids = np.array([tokens], dtype=np.int64)
        logits = session.run(None, {"input_ids": input_ids})[0]
        next_token = int(np.argmax(logits[0, -1, :]))
        tokens.append(next_token)
        if next_token == end_id:
            break

    return "".join([itos[t] for t in tokens])

print(run_mathscout("2 * (3 + 4 * 5)"))

6. Limitations and Experimental Boundaries

Lack of Operational Maturity: This model is a research proof of concept. It has not undergone safety alignment, red-teaming, or stress-testing under arbitrary adversarial inputs.
Strict Domain Constraint: The model is solely trained on symbolic math expressions. It has no natural language understanding, conversational dialogue ability, or general knowledge.
Format Sensitivity: The model requires strict adherence to token spacing rules; deviation in spacing can trigger reasoning failures.
Context Limit: Complex expressions exceeding 512 tokens in total scratchpad length will be truncated.

7. License

Released under the Apache License, Version 2.0. See the LICENSE file for full license text.

Downloads last month: -; Downloads are not tracked for this model. How to track