CMA-1M-Mini

Validation-selected checkpoint from a 0.96M-parameter Channel-Mixing Attention generalist language model using a deterministic 256-byte vocabulary plus four native control tokens. It has no arithmetic-specific token splitting, place embeddings, role embeddings, or inference-time equation detection. It was selected at step 10,000 for WikiText normalized BPB 1.7233. Its report-only Open SLM Leaderboard-style average is 33.81%.

Loading

This is a custom Transformers architecture. trust_remote_code=True is required because stock Hugging Face model classes do not implement CMA or this model's exact rotary convention.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "User01110/CMA-1M-Mini"
tokenizer = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    repo, trust_remote_code=True, dtype="auto"
)

Checkpoint tensors are stored in bfloat16. Pass dtype=torch.float32 when an FP32 runtime is required; every stored BF16 value widens exactly to FP32, though the pre-export FP32 master-weight mantissa cannot be reconstructed.

Architecture

• Parameters: 958,692, with tied input/output embeddings
• Weights: native bfloat16 safetensors (model.safetensors); no .bin weights
• Runtime: PyTorch 2.5+ for native SDPA grouped-query attention
• Tokenizer: deterministic local UTF-8 byte tokenizer (byte-v1)
• Vocabulary: 256 bytes plus PAD/BOS/EOS/UNK (260 total)
• Parameter allocation: 33,280 tied embedding parameters and 925,412 non-embedding parameters
• Training/validation context: 2,048 byte tokens
• Exported benchmark/inference context: 2,048 byte tokens
• Standalone prompt tokenization automatically prepends the native BOS token
• Width/layers: 128 / 6
• Token-attention heads: 4 query, 2 KV
• CMA: 8 slots of 16 channels, 2 routing heads, expansion 2
• Each token receives dense values followed by content-dependent softmax routing across channel slots
• Contiguous-half RoPE without scaling
• No task-specific model or tokenizer features

Training mixture

• FineWeb-Edu 100BT shuffled: 45.00% of trained tokens
• DCLM-Baseline 1.0: 25.00% of trained tokens
• DCLM-Edu: 10.00% of trained tokens
• Cosmopedia v2: 10.00% of trained tokens
• FineMath 4+: 10.00% of trained tokens

DCLM-Baseline adds broadly curated web diversity, DCLM-Edu adds an education-filtered DCLM view, and FineMath-4+ supplies mathematical explanations and reasoning as ordinary causal-language-model text. There are no task-specific model or tokenizer features. All five training sources are streamed natural-text corpora.

Zero-shot evaluation at step 10,000

The four lm-eval tasks use normalized accuracy when supplied by lm-eval 0.4.12, with native bfloat16 weights and float32 likelihood softmax. ArithMark uses the same precision policy and its official raw continuation log-likelihood-sum rule. Autocast is not used for evaluation. Every independent benchmark context starts with the native BOS token.

Benchmark	Accuracy
HellaSwag	28.99%
ARC-Easy	28.70%
ARC-Challenge	20.82%
PIQA	54.19%
ArithMark-2	27.32%
ARC mean	24.76%
Open SLM Leaderboard-style average	33.81%

The average is (HellaSwag + mean(ARC-Easy, ARC-Challenge) + PIQA + ArithMark-2) / 4, matching the Open SLM Leaderboard formula.

WikiText-103 validation at this step: loss 1.1945, perplexity 3.30, normalized BPB 1.7233 over 1,144,831 scored tokens and 1,144,831 normalized UTF-8 bytes, using one initial BOS, 2,048-token windows, and a 1,024-token stride.