Local-Axiom-AI/Sabaki-Preview

Sabaki Preview

Sabaki our first attempt at a small, experimental language model. It was trained from scratch with a custom architecture.

This upload is a research preview, not a polished assistant. It is here for people who want to inspect the design, run local experiments, or continue post-training. Expect rough edges, including uneven instruction following, repetition, factual mistakes, and sensitivity to prompts.

At a glance

Model	Decoder-only causal language model
Size	221.5M trainable parameters
Active per token	130.7M parameters, excluding the tied vocabulary matrix
Layers	16 total: 12 recurrent and 4 global-attention layers
Hidden size	768
Attention heads	12 x 64
Experts	8 routed experts (Top-2) and 2 shared experts
Vocabulary	49,152-token lossless byte-level BPE
Configured context	Up to 16,384 tokens
Pretraining	8.0B tokens
Post-training	About 23.9M assistant-target tokens
Formats	FP16 PyTorch checkpoint (.pt) and FP16 GGUF
License	CC BY-NC 4.0

The 16,384-token context is a configuration limit, not a promise of reliable recall across the full window. Training used a curriculum of 512, 2,048, and 4,096 tokens, so behavior beyond 4K has not been established.

Known issue: runaway generation

Sabaki has a serious tendency to answer a prompt and then continue generating unrelated or nonsensical text instead of stopping. In some cases it will keep going until it reaches the generation limit. This is a model behavior issue, not just a sampling quirk.

Always set a maximum token limit when running the preview, and do not rely on it to stop cleanly on its own. A repetition penalty can help, but it does not eliminate the problem.

How it works

Most layers use Kimi Delta Attention (KDA), which carries a fixed-size recurrent state forward as it reads. Every fourth layer uses Multi-head Latent Attention (MLA) with rotary position embeddings, giving the model periodic access to the full preceding context. This keeps most sequence processing linear while still allowing global attention.

The feed-forward side is sparse in all but the first block. For each token, the router selects two of eight routed experts and combines them with two shared experts. Persistent routing biases help keep expert use balanced without adding a separate balancing loss.

The complete layer schedule is:

Blocks	Sequence mixer	Feed-forward path
1	KDA	Dense SiTU-GLU, width 2,048
2-3	KDA	Top-2 routed MoE + 2 shared experts
4	Rotary MLA	Top-2 routed MoE + 2 shared experts
5-16	Three KDA blocks followed by one rotary MLA block, repeated	Top-2 routed MoE + 2 shared experts

Input and output embeddings are tied, and every block uses pre-normalization and ordinary residual connections.

Running the model

The model repository includes both the original FP16 PyTorch checkpoint (.pt) and an FP16 GGUF export. The GGUF file contains custom tensors and requires the Sabaki Preview llama.cpp runtime. It does not currently run in stock transformers or upstream llama.cpp.

Clone and build the companion runtime, then start its launcher with Sabaki-Preview-F16.gguf as the model. The launcher provides a browser UI and an OpenAI-compatible local API. See the runtime repository for platform-specific build and launch instructions.

For a simple conversation, use plain role labels:

USER: Explain why the sky appears blue.
ASSISTANT:

A conservative place to start is temperature 0.1, top-k 50, and repeat penalty 1.15. These are practical defaults, not universally optimal settings.

Evaluation

The table below comes from the nearest fully evaluated training snapshot. That snapshot had completed the same 8,000,045,056 pretraining tokens and 20,016,551 assistant-target post-training tokens. The weights in this upload are from a slightly later post-training checkpoint, so treat these numbers as a useful reference rather than exact scores for this file.

Benchmark	Questions	Zero-shot normalized accuracy
HellaSwag	10,042	31.34%
ARC-Easy	2,376	40.49%
ARC-Challenge	1,172	25.34%
PIQA	1,838	60.72%
ArithMark-3	1,000	36.60%

The evaluation used an unquantized FP16 export and a custom native likelihood evaluator. These are not independently verified leaderboard results. The prompts were zero-shot completion prompts rather than the chat format shown above. ArithMark-3 also has a documented middle-answer shortcut that should be kept in mind when reading its score.

Leaderboard context

Taken as the first end-to-end training run of a new model design, this is a strong result. The table below places Sabaki beside several familiar models from the Open SLM Leaderboard. It is a comparison, not a claimed leaderboard placement: the leaderboard requires fewer than 150M total parameters, while Sabaki has 221.5M total parameters and 130.7M active non-embedding parameters per token.

Model	Parameters	HellaSwag	ARC-Easy	ARC-Challenge	PIQA	ArithMark-3
Sabaki Preview	221.5M total / 130.7M active	31.34%	40.49%	25.34%	60.72%	36.60%
GPT-2	124M	31.26%	39.35%	22.35%	62.08%	35.70%
OPT-125M	125M	31.31%	40.28%	22.70%	62.24%	35.20%
GTM-v2-chat	119.5M	30.70%	40.49%	25.17%	63.98%	36.60%
SmolLM2-135M	135M	43.22%	58.63%	29.69%	68.44%	39.20%

Sabaki scores higher than GPT-2 and OPT-125M on four of these five benchmarks. Against GTM-v2-chat, it is ahead on HellaSwag and ARC-Challenge, tied on ARC-Easy and ArithMark-3, and behind on PIQA. SmolLM2-135M remains clearly stronger across the board and is a useful picture of how much headroom remains.

The training budget makes that result notable. Sabaki saw 8B pretraining tokens, compared with 180B reported for OPT and 2T for SmolLM2-135M, or about 22 times and 250 times as many, respectively. Its post-training was also small at roughly 23.9M assistant-target tokens; for context, GTM-v2-chat reports about 1B SFT token exposures over several passes. These are not perfectly controlled comparisons because model design, data quality, tokenization, training objectives, and evaluation implementations all differ. Even so, the results suggest the core design learned useful capabilities unusually early.

Good uses for this preview

Sabaki is a good fit for:

• studying compact recurrent/attention hybrids;
• experimenting with sparse mixture-of-experts models;
• local, non-commercial text-generation projects;
• evaluation and further post-training by researchers comfortable with custom runtimes.

It is not a good fit for autonomous decisions or situations where a wrong answer could cause medical, legal, financial, physical, or other serious harm. The model has not received comprehensive safety evaluation or preference optimization, and it can invent facts, citations, code, and tool results. English dominates its training data; other languages have not been evaluated separately.

License

The model weights are available under the Creative Commons Attribution-NonCommercial 4.0 International (CC BY-NC 4.0) license. Attribution is required, and commercial use is not permitted. Software and third-party datasets keep their own licenses; the model license does not replace those terms.

Citation

If Sabaki helps your research, please cite this model repository. A formal citation will be added with a future full release.

@software{sabaki_preview_2026,
  author = {{Local-Axiom-AI}},
  title  = {Sabaki Preview},
  year   = {2026},
  note   = {A compact hybrid recurrent, latent-attention, and sparse mixture-of-experts language model}
}