CMA-20M

Evaluated training checkpoint from a 20.21M-parameter Channel-Mixing Attention generalist language model using the third-party BananaMind 8,192-token digit-aware byte-level BPE tokenizer. It has no place embeddings, role embeddings, or inference-time equation detection. It was recorded at step 3,000 with WikiText normalized BPB 1.3915. Its report-only Open SLM Leaderboard-style average is 35.68%.

Loading

This is a custom Transformers architecture. trust_remote_code=True is required because stock Hugging Face model classes do not implement CMA or this model's exact rotary convention.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "User01110/CMA-20M"
tokenizer = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    repo, trust_remote_code=True, dtype="auto"
)

Checkpoint tensors are stored in bfloat16. Pass dtype=torch.float32 when an FP32 runtime is required; every stored BF16 value widens exactly to FP32, though the pre-export FP32 master-weight mantissa cannot be reconstructed.

Architecture

• Parameters: 20,212,355, with tied input/output embeddings
• Weights: native bfloat16 safetensors (model.safetensors); no .bin weights
• Runtime: PyTorch 2.5+ for native SDPA grouped-query attention
• Tokenizer: BananaMind/BananaMind-2-Mini at revision 84a0afb98db902caf07a1e949676d9fef9e5cf9e
• Vocabulary: 8,192 third-party tokens
• Parameter allocation: 2,752,512 tied embedding parameters and 17,459,843 non-embedding parameters
• Context: 1,024 tokens
• Standalone prompt tokenization automatically prepends the native BOS token
• Width/layers: 336 / 13
• Token-attention heads: 6 query, 2 KV
• CMA: 14 slots of 24 channels, 3 routing heads, expansion 3
• Each token receives dense values followed by content-dependent softmax routing across channel slots
• Signed routing uses 0.95 * softsign(signal), preserving at least a 5% base path under positive routing
• Contiguous-half RoPE without scaling
• No task-specific model features or inference-time benchmark handling

Tokenizer provenance

The tokenizer and its 8,192-token vocabulary were not created or owned by the CMA model author. They are reused from the public BananaMind/BananaMind-2-Mini repository at the exact revision listed above, whose repository metadata identifies BananaMind as the publisher and Apache-2.0 as the license. No claim of tokenizer ownership beyond that public attribution is made here. The exported copy preserves its vocabulary, merges, normalization, and digit-aware pre-tokenization; CMA only configures the existing BOS token to be prepended automatically and sets the model context length.

Optimization

• Training budget: 10,485,760,000 tokens over 20,000 updates
• Effective batch: 524,288 tokens per update
• Learning rate: 500-update linear warmup to 1.0e-03, then linear decay
• The final configured update is positive; update 20,001 is exactly zero
• Official PyTorch Muon with match_rms_adamw for hidden matrices; AdamW for embeddings and remaining parameters

Training mixture

• FineWeb-Edu 100BT shuffled: 45.00% of trained tokens
• DCLM-Baseline 1.0: 25.00% of trained tokens
• DCLM-Edu: 10.00% of trained tokens
• Cosmopedia v2: 10.00% of trained tokens
• FineMath 4+: 10.00% of trained tokens

DCLM-Baseline adds broadly curated web diversity, DCLM-Edu adds an education-filtered DCLM view, and FineMath-4+ supplies mathematical explanations and reasoning as ordinary causal-language-model text. There are no benchmark labels or benchmark-specific preprocessing. All five training sources are streamed natural-text corpora.

Zero-shot evaluation at step 3,000

The four lm-eval tasks use normalized accuracy when supplied by lm-eval 0.4.12, with native bfloat16 weights and float32 likelihood softmax. ArithMark uses the same precision policy and its official raw continuation log-likelihood-sum rule. Autocast is not used for evaluation. Every independent benchmark context starts with the native BOS token.

Benchmark	Accuracy
HellaSwag	27.53%
ARC-Easy	36.66%
ARC-Challenge	22.10%
PIQA	57.51%
ArithMark-2	28.28%
ARC mean	29.38%
Open SLM Leaderboard-style average	35.68%

The average is (HellaSwag + mean(ARC-Easy, ARC-Challenge) + PIQA + ArithMark-2) / 4, matching the Open SLM Leaderboard formula.

WikiText-103 validation at this step: loss 3.3821, perplexity 29.43, normalized BPB 1.3915 over 326,337 scored tokens and 1,144,346 normalized UTF-8 bytes, using one initial BOS, 1,024-token windows, and a 1,024-token stride.