OxMini

OxMini is a 2.17M-parameter character language model trained as a transparent, CPU-scale architecture experiment.

Architecture honesty notice: OxMini is not “Ox Alpha shrunk down,” is not GLM-5.3-Flash, and is not a reproduction of either system. Public reporting identifies Ox Alpha as GLM-5.3-Flash, a 320B-total/18B-active sparse MoE with KDA, MLA+DSA, four-stream mHC, and vision. OxMini only studies toy approximations of the 3:1 hybrid pattern and multi-stream residual routing. It has no MoE, DSA indexer, vision encoder, long-context kernel, or comparable capability.

Architecture

Source component	OxMini approximation	Status
3:1 KDA : MLA/DSA attention	Six recurrent KDA-lite + two global MLA-lite blocks	Simplified
KDA	Explicit channel-gated delta-rule recurrence, NoPE	Simplified; no chunk kernel
MLA	Full queries plus low-rank joint K/V bottleneck and RoPE	Simplified
DSA	None	Dropped
Sparse MoE	Dense SwiGLU	Dropped
Four-stream mHC	Static four-stream softmax routing; optional Sinkhorn projection	Substantially simplified
Vision	None	Dropped

• Parameters: 2,169,388
• Context: 256 characters
• Layers: 8 (KDA-lite ×3, MLA-lite ×1, repeated)
• Width / heads: 128 / 4
• Residual streams: 4
• Tokenizer: corpus-built character vocabulary with <unk>
• Weight format: safetensors

Training data

The checkpoint uses Tiny Shakespeare downloaded from the Karpathy char-rnn corpus URL. It is deterministically split 90%/5%/5% by character into train/validation/test, and the vocabulary is built only from the training split. Shakespeare's underlying plays are public-domain text; this model repository does not redistribute prepared corpus files and does not claim ownership over source text.

• Corpus: 1,115,394 characters; SHA-256 86c4e6aa9db7c042ec79f339dcb96d42b0075e16b8fc2e86bf0ca57e2dc565ed
• Splits: 1,003,854 train / 55,770 validation / 55,770 test characters
• Vocabulary: 66 tokens including <unk>; zero unknown tokens in all splits
• Optimizer: AdamW with 5% warmup, cosine decay, and gradient clipping
• Run: 1,000 updates, batch size 8, seed 1337, CPU only
• Hardware/time: Apple M4 Pro, 10 threads, PyTorch 2.9.1, 1,820.9 seconds

Measured evaluation and ablation

Release checkpoint

Metric	Value
Initial validation loss	4.2799
Final/best validation loss	1.7944
Test cross-entropy	1.80425
Test perplexity	6.075
Test next-character accuracy	47.79%

The seen/unseen copy probe was inconclusive: seen accuracy was higher (50.0% vs 37.5%), but seen cross-entropy was worse (2.213 vs 2.075). The arithmetic probe was skipped because the vocabulary does not contain all digits.

Matched 120-step ablation

Variant	Parameters	Validation loss	Test PPL	Next-char accuracy	Time
Plain GPT	2,107,776	2.8228	16.54	26.17%	17.1s
Hybrid, one stream	2,168,984	2.4979	12.02	37.35%	150.2s
Hybrid + mHC-lite	2,169,388	2.5595	12.78	35.21%	153.5s

All values are copied from local JSON/CSV artifacts included in this repository. The ablation arms used the same data, seed, update count, context, width/depth, and training recipe; parameter spread was 2.84%. Hybrid-only did best in this short run; mHC-lite did not help. The recurrent arms were about nine times slower than baseline because this implementation prioritizes a legible Python recurrence over a chunk-parallel kernel. These toy, single-seed results are descriptive and do not establish that KDA or mHC is generally better.

Sample generations

Temperature 0.7, 120 new characters, unedited:

ROMEO:
He prove sword in Recadars, he with that my grow say the broble joy these dead
The words.

WARWICK:
King to have done's

The sample learns speaker labels, line breaks, and some Shakespeare-like surface form, but remains incoherent. See samples.md for every fixed prompt at temperatures 0.7 and 1.0; none were filtered or cherry-picked.

Load

This is a plain PyTorch custom architecture, not a Transformers AutoModel package. Download the repository and import its bundled source:

from pathlib import Path
import sys
from huggingface_hub import snapshot_download

snapshot = Path(snapshot_download("Shivam3002/OxMini"))
sys.path.insert(0, str(snapshot / "src"))

from oxmini import CharTokenizer, OxMiniForCausalLM
from oxmini.generate import generate_text

model = OxMiniForCausalLM.from_pretrained(snapshot).eval()
tokenizer = CharTokenizer.load(snapshot / "tokenizer.json")
print(generate_text(model, tokenizer, "ROMEO:\n", max_new_tokens=120, temperature=0.8))

Intended use

Educational inspection, CPU smoke tests, and toy ablations only.

Limitations

This model is not suitable for factual, safety-critical, production, or user-facing tasks. Its corpus is tiny and narrow, its 256-character context is short, and character-level samples can be incoherent. The probes are diagnostic rather than standard language-model benchmarks. Architecture labels ending in “-lite” are approximations, not faithful kernels.

References

• Sebastian Raschka, GLM-5.3-Flash Architecture Notes, Aug. 26, 2026.
• Kimi Team et al., Kimi Linear and official code.
• DeepSeek-AI, DeepSeek-V3.2 and V3.2-Exp code.
• Xie et al., mHC.
• Hugging Face, DeepSeek-V4 documentation.
• Zhu et al., Hyper-Connections.

License

MIT for OxMini code and original model artifacts. Source-text rights remain with their respective sources.