Sabaki Preview
Sabaki is an experimental decoder-only language model implemented in PyTorch and our current best fully custom model(as of 10/08/2026). It combines recurrent Kimi Delta Attention (KDA), periodic gated rotary attention, and dense SiTU-GLU feed-forward layers.
This post-trained model is a research preview for local experiments, evaluation, and further post-training. Instruction following and factual accuracy can be uneven.
At a glance
| Property | Value |
|---|---|
| Model's parameters per token, excluding embeddings | 124,128,144 (124.1M); dense feed-forward layers throughout |
| Layers | 16: 12 recurrent KDA layers and 4 global-attention layers |
| Hidden size | 768 |
| Heads | 12 heads of width 64 |
| Feed-forward layers | Dense SiTU-GLU, intermediate width 2,048 in every block |
| Vocabulary | 49,152-token lossless byte-level BPE |
| Configured context | 16,384 tokens |
| Implementation | Custom PyTorch model with FP32 weights; BF16 training compute |
| License | CC BY-NC 4.0 |
The 16,384-token context is a configuration limit, not a measured long-context capability. The pretraining continuation used 2,048-token sequences, and the post-training checkpoints used 4,096-token sequences. Reliable behavior beyond those training lengths has not been established.
Model design
The model repeats three KDA blocks followed by one gated rotary-attention block:
| Blocks (one-based) | Sequence mixer | Feed-forward path |
|---|---|---|
| 1-3, 5-7, 9-11, 13-15 | Kimi Delta Attention | Dense SiTU-GLU, width 2,048 |
| 4, 8, 12, 16 | Gated causal multi-head attention with RoPE | Dense SiTU-GLU, width 2,048 |
Each block applies RMSNorm before its sequence mixer and before its feed-forward layer, with an ordinary residual addition after each. A final RMSNorm precedes the tied vocabulary projection. RMSNorm epsilon is 1e-6.
Recurrent KDA layers
KDA projects queries, keys, and values at the full 768-channel width and applies separate causal depthwise convolutions with kernel size 4. Queries and keys are normalized before the delta-rule recurrence. A rank-64 projection produces input-dependent per-channel decay, and a sigmoid write gate controls state updates. Each head maintains a 64 x 64 recurrent matrix, plus short convolution histories. The recurrent output is normalized, multiplied by a learned sigmoid output gate, and projected back to the residual stream.
These layers carry fixed-size state as the sequence grows. The four global-attention layers still retain a conventional KV cache, so the complete model does not have a constant-size cache.
Gated rotary attention
The global layers use full-width Q, K, and V projections with 12 query/key/value heads of width 64. RoPE is applied to Q and K with base 10,000, followed by causal scaled dot-product attention. A learned sigmoid gate modulates the merged heads before the output projection. These are standard full-head attention layers, not latent-compressed MLA.
Dense SiTU-GLU feed-forward layers
Every block projects the normalized hidden state into two 2,048-wide branches, applies the bounded SiTU gate and up activations, multiplies them, and projects back to width 768. With gate preactivation g and up preactivation u, the activation is:
(4 * tanh(g / 4) * sigmoid(g)) * (25 * tanh(u / 25))
There are no routed experts, shared experts, routers, or expert-balancing biases. The dense checkpoints are incompatible with the older MoE architecture.
| Component | Parameters, excluding embeddings |
|---|---|
| 12 KDA mixers, including their internal normalization | 36,808,848 |
| 4 gated rotary-attention mixers | 11,796,480 |
| 16 dense feed-forward layers | 75,497,472 |
| Block and final RMSNorm scales | 25,344 |
| Total | 124,128,144 |
Training
| Stage | Training tokens |
|---|---|
| Pretraining | 16B tokens |
| Supervised fine-tuning (SFT) | About 200M assistant-target tokens |
| Further grounded-chat SFT | About 165K assistant-target tokens |
The completed checkpoints contain FP32 model tensors and no optimizer state. Training used BF16 compute.
Running with PyTorch
Use Sabaki's custom PyTorch implementation with the matching tokenizer and trained weights. Generation runs the model in evaluation mode with gradients disabled, processes the prompt once, and then uses cached recurrent states and attention keys/values to generate subsequent tokens.
The prompt serializer uses dedicated control-token IDs. A single-turn prompt has this structure:
<|bos|><|user|>Explain why the sky appears blue.<|user_end|><|assistant|>
Assistant answers use <|final|>...<|final_end|><|assistant_end|>, with <|eos|> closing a training episode. Use the tokenizer's control-token serialization to construct this prompt. Plain USER: / ASSISTANT: labels are not the trained protocol.
The PyTorch generation defaults are temperature 0.2, top-p 0.95, and a maximum of 128 new tokens. Generation stops at <|final_end|> or <|eos|>, or when the token budget is exhausted. Keep the prompt and generated continuation within the configured context length.
Always set a generation limit.
Evaluation
These scores describe Sabaki after the 200M-token SFT stage. The later grounded chat stage has a separate evaluation below.
| Benchmark | Questions | Zero-shot normalized accuracy |
|---|---|---|
| HellaSwag | 10,042 | 36.85% |
| ARC-Easy | 2,376 | 47.05% |
| ARC-Challenge | 1,172 | 28.75% |
| PIQA | 1,838 | 65.83% |
| ArithMark-3 | 1,000 | 36.00% |
These are local evaluation results, not a verified leaderboard submission.
The later grounded-chat checkpoint has a separate assistant-token evaluation: held-out NLL 2.4488 and perplexity 11.5746 across 50 conversations and 3,703 assistant tokens. This does not replace a five-benchmark evaluation of those weights.
Intelligence Index comparison
Sabaki scores 20.33 on the Open SLM Leaderboard's Intelligence Index (INT), using the full benchmark evaluation of the 200M token SFT stage. The index adjusts each benchmark for chance performance, averages ARC-Easy and ARC-Challenge into a single ARC component, and weights HellaSwag, ARC, and PIQA at 1.0 each and ArithMark-3 at 0.65.
From our own testing of models on the leaderboard we have found that many of the models do very poorly on real world questions and tasks when post-trained. This observation has been seen on different model designs, ruling out the possibility of design choices producing this outcome. The most likely conclusion is that the other models were not built for real use and were trained on data that would help the benchmark rather than produce a overall rounded pre/post trained model. We are not accusing other groups of benchmark inflation or contaminated training data, just pointing out the diffrences in our approaches.
The following comparison uses the Open SLM Leaderboard scores and formula checked on 12 October 2026. Higher is better.
| Model | INT score |
|---|---|
| GPT-X3-150M | 33.05 |
| SmolLM2-135M | 27.13 |
| MobileLLM-R1-140M-base | 24.64 |
| GPT-X2-125M | 23.36 |
| JugnuLM-110M-R2+ | 20.39 |
| Sabaki | 20.33 |
| Speck2-140M | 20.02 |
| GPT-X-125M | 19.94 |
| GTM-v2-chat | 15.36 |
| OPT-125M | 13.80 |
| GPT-2 | 13.58 |
Sabaki is numerically close to JugnuLM-110M-R2+, slightly ahead of Speck2-140M and GPT-X-125M, and above GPT-2 and OPT-125M. It trails SmolLM2-135M by 6.80 INT points. Small differences should not be interpreted as established performance advantages.
By score alone, Sabaki would sit 21st among the 210 existing entries with all five benchmark scores checked on that date. This is an unofficial comparison using local PyTorch results; it does not establish leaderboard eligibility or a verified placement. The later grounded-chat checkpoint has no recorded full INT evaluation.
Limitations
This preview can produce incorrect facts, invented citations, repetitive text, and unreliable instructions. Languages other than English have not been evaluated separately. Protocol tokens for tools and other structured outputs do not establish reliable tool-use capabilities. The model has not received comprehensive safety evaluation or preference optimization.
Future
Our approch to this group of models is to create a strong pretrained/postrained model that can be easily adapted for uses that require long context and precise recalling of information, this requires more of the model to be dedicated to attention. A variant of this model is currently being trained with less of the parameters being alocated to attention and vocab, however this model will see less tokens during the training so comparison may not be representative.
License
The model weights are available under the Creative Commons Attribution-NonCommercial 4.0 International (CC BY-NC 4.0) license. Attribution is required, and commercial use is not permitted. Software retains its own license.
Citation
If this preview helps your research, please cite the model repository. A formal citation can be added with a full release.
@software{sabaki_preview_2026,
author = {{Local-Axiom-AI}},
title = {Sabaki Preview},
year = {2026},
note = {A compact hybrid recurrent and gated rotary-attention language model with dense SiTU feed-forward layers}
}