CMA2-8M

Benchmark-selected checkpoint from a 7.71M-parameter Channel-Mixing Attention generalist language model using the native GPT-S 4,096-token vocabulary. It has no arithmetic-specific token splitting, place embeddings, role embeddings, or inference-time equation detection. It was selected at step 10,000 for an Open SLM Leaderboard-style average of 34.38%. WikiText normalized BPB is reported separately and is not used for selection.

Loading

This is a custom Transformers architecture. trust_remote_code=True is required because stock Hugging Face model classes do not implement CMA or this model's exact rotary convention.

from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "User01110/CMA2-8M"
tokenizer = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    repo, trust_remote_code=True, dtype="auto"
)

Architecture

• Parameters: 7,712,418, with tied input/output embeddings
• Weights: safetensors only (model.safetensors); no PyTorch .bin weights
• Runtime: PyTorch 2.5+ for native SDPA grouped-query attention
• Tokenizer: AxiomicLabs/GPT-S-5M at revision df47402
• Vocabulary: 4,096 native tokens
• Parameter allocation: 983,040 tied embedding parameters and 6,729,378 non-embedding parameters
• Context: 2,048 tokens
• Standalone prompt tokenization automatically prepends the native BOS token
• Width/layers: 240 / 13
• Token-attention heads: 6 query, 2 KV
• CMA: chunk=24, heads=3, expansion=2
• Contiguous-half RoPE without scaling
• No task-specific model or tokenizer features

Training mixture

• FineWeb-Edu 100BT shuffled: 60.00% of trained tokens
• Cosmopedia v2: 15.00% of trained tokens
• FineWeb-HQ: 15.00% of trained tokens
• FineMath 4+: 10.00% of trained tokens

FineMath-4+ supplies high-quality mathematical explanations and reasoning as ordinary causal-language-model text, with no task-specific model features or tokenizer features. All four training sources are streamed natural-text corpora.

Zero-shot evaluation at step 10,000

The four lm-eval tasks use normalized accuracy when supplied by lm-eval 0.4.12, with float32 weights and softmax. ArithMark uses float32 weights and its official raw continuation log-likelihood-sum rule. Every independent benchmark context starts with the native BOS token.

Benchmark	Accuracy
HellaSwag	27.59%
ARC-Easy	32.95%
ARC-Challenge	21.67%
PIQA	55.71%
ArithMark-2	26.88%
ARC mean	27.31%
Open SLM Leaderboard-style average	34.38%

The average is (HellaSwag + mean(ARC-Easy, ARC-Challenge) + PIQA + ArithMark-2) / 4, matching the Open SLM Leaderboard formula.

WikiText-103 validation at this step: loss 3.1430, perplexity 23.17, normalized BPB 1.4210 over 358,399 scored tokens and 1,143,646 normalized UTF-8 bytes, using one initial BOS, 2,048-token windows, and a 1,024-token stride.