CMA3

Evaluated training checkpoint from a 19.58M-parameter Post-CMA (post-conditioned Channel-Mixing Attention) generalist language model using the third-party BananaMind 8,192-token digit-aware byte-level BPE tokenizer. It has no place embeddings, role embeddings, or inference-time equation detection. It was recorded at step 1,000 with WikiText normalized BPB 1.4339.

Loading

This is a custom Transformers architecture. trust_remote_code=True is required because stock Hugging Face model classes do not implement CMA or this model's exact rotary convention.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "User01110/cma3"
tokenizer = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    repo, trust_remote_code=True, dtype="auto"
)

Checkpoint tensors are stored in bfloat16. Pass dtype=torch.float32 when an FP32 runtime is required; every stored BF16 value widens exactly to FP32, though the pre-export FP32 master-weight mantissa cannot be reconstructed.

Architecture

• Parameters: 19,576,524, with tied input/output embeddings
• Weights: native bfloat16 safetensors (model.safetensors); no .bin weights
• Runtime: PyTorch 2.5+ native SDPA for token attention and compiled Post-CMA routing
• Tokenizer: BananaMind/BananaMind-2-Mini at revision 84a0afb98db902caf07a1e949676d9fef9e5cf9e
• Vocabulary: 8,192 third-party tokens
• Parameter allocation: 2,457,600 tied embedding parameters and 17,118,924 non-embedding parameters
• Context: 1,024 tokens
• Standalone prompt tokenization automatically prepends the native BOS token
• Width/layers: 300 / 16
• Token-attention heads: 6 query, 2 KV
• CMA: 15 slots of 20 channels, 4 routing heads, expansion 3
• Ordering: Post-CMA (SwiGLU activation -> channel routing -> output projection)
• Each source slot first forms a complete SwiGLU feature, u = W_v x * SiLU(W_g x)
• Content-dependent softmax routing mixes those completed features across channel slots, then W_o projects the result
• CMA has no scalar route gate or fallback blend: CMA(x) = W_o(A(x, u) u)
• Contiguous-half RoPE without scaling
• No task-specific model features or inference-time benchmark handling

Tokenizer provenance

The tokenizer and its 8,192-token vocabulary were not created or owned by the CMA model author. They are reused from the public BananaMind/BananaMind-2-Mini repository at the exact revision listed above, whose repository metadata identifies BananaMind as the publisher and Apache-2.0 as the license. No claim of tokenizer ownership beyond that public attribution is made here. The exported copy preserves its vocabulary, merges, normalization, and digit-aware pre-tokenization; CMA only configures the existing BOS token to be prepended automatically and sets the model context length.

Optimization

• Training budget: 15,728,640,000 tokens over 30,000 updates
• Effective batch: 524,288 tokens per update
• Memory-safe microbatch: 128 sequences x 1,024 tokens, accumulated 4 times; CMA routing-score activations are recomputed during backward without changing the model function
• Runtime split: 2 deterministic, disjoint interleaved-file data shards of 15,000 updates; full-precision optimizer/RNG/data-stream continuation
• Learning rate: 1,000-update linear warmup to 2.0e-03; hold through 9,000; linear transition to 1.0e-03 by 10,000; hold through 20,000; then cosine decay from 1.0e-03 toward zero
• The final configured update is positive; update 30,001 is exactly zero
• Official PyTorch Muon with match_rms_adamw for hidden matrices; AdamW for embeddings and remaining parameters

Training mixture

• FineWeb-Edu 100BT shuffled: 30%
• Ultra-FineWeb English: 15%
• DCLM-Edu score 3+ prefiltered: 20%
• Wikipedia English: 15%
• Cosmopedia v2: 10%
• FineMath 4+: 10%

FineWeb-Edu, Ultra-FineWeb, and DCLM-Edu contribute complementary filtered general and educational web text, Wikipedia adds grounded encyclopedic material, Cosmopedia supplies synthetic explanations, and FineMath-4+ supplies mathematical reasoning as ordinary causal-language-model text. The mixture remains fixed for the complete run. There are no benchmark labels or benchmark-specific preprocessing. Dataset revisions are pinned and the two runtime sessions receive alternating physical files from every source, avoiding a contiguous-half distribution transition.

Zero-shot evaluation at step 1,000

The four lm-eval tasks use normalized accuracy when supplied by lm-eval 0.4.12, with native bfloat16 weights and float32 likelihood softmax. ArithMark-3 uses its official primary acc_norm metric: mean continuation-token log likelihood, with context and continuation tokenized separately after one native BOS prefix. Autocast is not used for evaluation. The four lm-eval benchmark contexts use the tokenizer's native BOS behavior.

Open SLM Intelligence Index: 6.62 Open SLM Average: 35.66%

Benchmark	Accuracy
HellaSwag	27.43%
ARC-Easy	34.34%
ARC-Challenge	20.90%
PIQA	55.50%
ArithMark-3 (acc_norm)	32.10%

The Average equally weights HellaSwag, combined ARC (the mean of ARC-Easy and ARC-Challenge), PIQA, and ArithMark-3. The Intelligence Index first maps random chance to 0 and perfect accuracy to 100, then weights HellaSwag, combined ARC, and PIQA at 1.0 and ArithMark-3 at 0.75, matching Open SLM Leaderboard revision b7781f0efb9df61912f94cf0e196ed32becded83.

WikiText-103 validation at this step: loss 3.4852, perplexity 32.63, normalized BPB 1.4339 over 326,703 scored tokens and 1,145,591 normalized UTF-8 bytes, using one initial BOS, 1,024-token windows, and a 512-token stride.