User01110/CMA-1M-Mini

CMA-1M Mini

Experimental research model, generations are fully unreliable. CMA-1M Mini exists only as a test bed for understanding how Channel-Mixing Attention works, where it helps, and where its limits and failure modes appear. Do not treat its output as factual, safe, coherent, or suitable for production or real-world decisions.

CMA-1M Mini is a 958,692-parameter base causal language model built to test Channel-Mixing Attention (CMA) at very small scale. It combines causal grouped-query token attention with content-dependent mixing across each token's hidden channels. The model uses a lossless byte-level tokenizer, tied embeddings, native BF16 weights, and a 2,048-token context window.

Parameters	958,692
Architecture	Decoder-only CMA causal LM
Context	2,048 byte tokens
Vocabulary	260 tokens: 256 bytes + PAD/BOS/EOS/UNK
Weight format	BF16 Safetensors
Intended interface	Plain-text completion

This is a pretrained base model, not a chat or instruction model. Give it ordinary text to continue; no chat template or role markers are required.

Quick start

The architecture is provided as custom Transformers code, so trust_remote_code=True is required. PyTorch 2.5 or newer is recommended.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo_id = "User01110/CMA-1M-Mini"
device = "cuda" if torch.cuda.is_available() else "cpu"
dtype = (
    torch.bfloat16
    if device == "cuda" and torch.cuda.is_bf16_supported()
    else torch.float32
)

tokenizer = AutoTokenizer.from_pretrained(
    repo_id,
    trust_remote_code=True,
)
model = AutoModelForCausalLM.from_pretrained(
    repo_id,
    trust_remote_code=True,
    dtype=dtype,
).to(device).eval()

prompt = "The process of photosynthesis"
inputs = tokenizer(prompt, return_tensors="pt")
inputs = {name: tensor.to(device) for name, tensor in inputs.items()}

with torch.inference_mode():
    output = model.generate(
        **inputs,
        max_new_tokens=96,
        do_sample=False,
    )

print(tokenizer.decode(output[0], skip_special_tokens=True))

The tokenizer automatically prepends <bos> during normal encoding. It does not append <eos> to a prompt; generation ends when the model emits EOS or reaches the requested length. If you deliberately use add_special_tokens=False, prepend BOS yourself.

Generation options

The included generation defaults are deterministic decoding with a repetition penalty of 1.2. Override them per request as needed.

Goal	Recommended settings
Reproducible completion	do_sample=False
Balanced sampling	do_sample=True, temperature=0.8, top_p=0.9, top_k=50
More varied text	do_sample=True, temperature=1.0, top_p=0.95
Reduce loops	repetition_penalty=1.1 to 1.2
Beam search	do_sample=False, num_beams=4
Output length	Set max_new_tokens; keep prompt + output within 2,048 tokens

Example with sampling:

with torch.inference_mode():
    output = model.generate(
        **inputs,
        max_new_tokens=128,
        do_sample=True,
        temperature=0.8,
        top_p=0.9,
        top_k=50,
        repetition_penalty=1.15,
    )

For the high-level pipeline API:

import torch
from transformers import pipeline

generator = pipeline(
    "text-generation",
    model="User01110/CMA-1M-Mini",
    tokenizer="User01110/CMA-1M-Mini",
    trust_remote_code=True,
    dtype="auto",
    device=0 if torch.cuda.is_available() else -1,
)

result = generator(
    "In a distant future,",
    max_new_tokens=80,
    do_sample=True,
    temperature=0.8,
    top_p=0.9,
)
print(result[0]["generated_text"])

To score text rather than generate it:

encoded = tokenizer("CMA models text one byte at a time.", return_tensors="pt")
encoded = {name: tensor.to(device) for name, tensor in encoded.items()}

with torch.inference_mode():
    result = model(**encoded, labels=encoded["input_ids"])

print(float(result.loss))

Tokenizer and context

• Every UTF-8 byte has a token, so ordinary text cannot become out-of-vocabulary.
• The four control tokens are <pad> (0), <bos> (1), <eos> (2), and <unk> (3).
• The context limit is 2,048 byte tokens, not 2,048 words or subword tokens.
• Non-ASCII text usually consumes multiple byte tokens per character.
• For long inputs, explicitly keep the most recent 2,048 tokens rather than relying on implicit truncation.
• The tokenizer has no arithmetic-specific splitting, chat template, or hidden prompt transformation.

Architecture

Component	Configuration
Hidden width / layers	128 / 6
Token attention	4 query heads, 2 key-value heads
Position encoding	Contiguous-half rotary embeddings, no scaling
CMA layout	8 channel slots x 16 channels
CMA routing	2 heads, expansion 2, content-dependent softmax mixing
CMA initialization	90% diagonal routing prior with a dense base path
Feed-forward gate	SiLU-gated routed values
Normalization	RMSNorm
Embeddings	Input and output weights tied
Attention runtime	Native PyTorch scaled-dot-product attention

For each token, CMA projects dense values, reshapes them into channel slots, and learns a softmax mixing matrix between those slots. A bounded learned coefficient controls the routed difference from the dense base value, so routing enriches rather than replaces the fallback path.

The exported generation implementation does not maintain a KV cache. This keeps the custom model compact and straightforward, but long autoregressive generations will recompute the active context and are slower than cached generation.

Training data

The model was pretrained as a general causal language model on the following mixture. Percentages describe the trained-token mixture.

Source	Share
FineWeb-Edu 100BT	45%
DCLM-Baseline 1.0	25%
DCLM-Edu	10%
Cosmopedia v2	10%
FineMath 4+	10%

No benchmark-specific prompts, task detectors, arithmetic vocabulary, or inference-time answer rules are built into the model.

Evaluation

Evaluation is zero-shot. The four lm-eval tasks use normalized accuracy when provided by lm-eval 0.4.12. ArithMark-2 uses raw continuation log-likelihood sums. Weights are evaluated in BF16 with FP32 likelihood softmax and an automatic BOS prefix.

Benchmark	Accuracy
HellaSwag	29.35%
ARC-Easy	29.29%
ARC-Challenge	21.76%
PIQA	54.62%
ArithMark-2	27.44%
Open SLM Leaderboard-style average	34.23%

The combined score is (HellaSwag + mean(ARC-Easy, ARC-Challenge) + PIQA + ArithMark-2) / 4. This is a report-only reproduction of the leaderboard formula, not an official leaderboard submission. WikiText-103 normalized validation BPB is 1.6974. Exact machine-readable results are available in benchmark_results.json.

Intended use and limitations

CMA-1M Mini is intended for architecture research, educational experiments, lightweight language-model tooling, and controlled comparisons at tiny scale.

• At fewer than one million parameters, generations are short-range and frequently incoherent; the model should not be treated as a knowledge source.
• It is not instruction-tuned, conversationally aligned, tool-using, or safety-tuned.
• Training data is predominantly English even though byte tokenization can represent any UTF-8 text.
• Outputs may reproduce biases, inaccuracies, or undesirable patterns from public training corpora.
• Do not use it for consequential medical, legal, financial, or safety decisions.
• Loading custom code executes files from the repository. Review the code or pin a trusted revision in security-sensitive environments.

Repository contents

• model.safetensors — BF16 model weights
• modeling_cma.py — Transformers-compatible CMA implementation
• config.json and generation_config.json — architecture and decoding defaults
• tokenizer.json and tokenizer_config.json — deterministic byte tokenizer
• benchmark_results.json — exact evaluation metrics and protocol metadata
• training_config.json — reproducibility configuration