Senryu-2-1B

Hybrid Gated DeltaNet-2 + Parallax chat model (1.09B) with 2.17B parameters of n-gram memory

By KerzgrAI: a 28-layer hybrid trained from random weights on 2.5B tokens of reasoning, tool-call and chat data

---

What this is

Senryu-2-1B is the first model on the Senryu-2 architecture and the first release from KerzgrAI. It succeeds Senryu (803M).

• 1.09B core parameters, embeddings included, plus 2.17B parameters of hashed 2- and 3-gram memory, read by lookup rather than matrix multiplies.
• No pretraining stage. It trained from random weights directly on chat data, at a 32,768-token sequence length on one RTX PRO 6000 Blackwell GPU.
• Thinking on or off, and Hermes JSON tool calls.

Try it in the demo Space.

---

Chat contract

Each assistant turn starts with a control token:

Mode	Prefix	Typical body
think	<|think|>\n	<think>…</think> then the answer
no-think	<|no_think|>\n	answer only

Tool schemas go in a Hermes-style <tools> block in the system turn, and the model calls them with:

<tool_call>
{"name": "web-search", "arguments": {"query": "…"}}
</tool_call>

Each observation comes back as a tool turn starting with <|tool_response|>\n.

---

Install & run

pip install torch safetensors tokenizers huggingface_hub
hf download KerzgrAI/Senryu-2-1B inference.py --local-dir .
python inference.py

python inference.py --prompt "Explain recursion in two sentences."
python inference.py --tools web-search,python,calculator
python inference.py --no-think --prompt "Reply in one sentence."
python inference.py --ngram-device cpu

inference.py downloads the weights, n-gram tables, tokenizer and tiny_gdn/, and installs flash-linear-attention from git at the pinned commit, so Git must be on PATH. A CUDA GPU is required: Gated DeltaNet-2 and Parallax run on Triton kernels.

Flag	Default	Description
--prompt	—	One-shot user message
--system	—	System prompt, used verbatim
--think / --no-think	think	Assistant control prefix
--tools	—	Built-ins: web-search, python, calculator (comma-separated)
--tools-json	—	Extra tool schemas (JSON list) for the <tools> block
--temperature	0.6	Sampling temperature (0.3 suits tool-calling turns)
--top-p	0.95	Nucleus sampling
--top-k	50	Top-k sampling
--repetition-penalty	1.08	Over the last --repetition-window (256) tokens
--max-new-tokens	4096	Max generation length
--context-length	32768	Conversation tokens kept (oldest dropped first)
--device	cuda	CUDA device, e.g. cuda:1
--ngram-device	same as --device	Where the n-gram tables live; cpu saves 4.3 GB of VRAM

Interactive commands: /think /no_think /system … /reset /exit. Ctrl+C stops the current reply.

---

Model architecture

Core parameters	1,094,961,660 (embeddings included)
N-gram memory	2,166,398,464 table parameters: 2- and 3-gram hashes, four hash heads per order
Layers	28: 7 × (3 Gated DeltaNet-2 + 1 Parallax attention)
Hidden	1,536
MLP	SiTU-GLU 3,584
Attention	Parallax: 12 query and 4 KV heads of 128, RoPE on half of each head (θ = 10⁶)
Linear	Gated DeltaNet-2, 12 heads × 128, with a short convolution
Residuals	Four-branch gated residual stream (rank-128 gate)
Memory read	Before block 1, with a context-aware gate per residual branch, as in Engram
Vocab	65,536 BPE, tied embeddings
Context	32,768

---

Training

From random weights, with no pretraining, on a fully shuffled mixture of the datasets listed in this card's metadata: 16 sequences of 32,768 tokens per step, Muon v2 + AdamW at 5×10⁻⁴, and sparse Adam at 2.5×10⁻³ for the n-gram tables. About 2.5B tokens in 56 hours. sft_config.json holds the run's settings.

---

Files

File	Contents
model.safetensors	Core weights, bfloat16
ngram_tables.safetensors	N-gram tables (bfloat16), canonical ids and hash multipliers
config.json	Architecture
tokenizer.json, special_token_ids.json	Tokenizer and control-token ids
inference.py, requirements.txt, tiny_gdn/	Standalone chat CLI and the model code it imports
sft_config.json	Training settings
windows_fla_patches/	Import patches inference.py applies to fla on Windows

---

Limitations

• Scale. 1.09B core parameters and 2.5B training tokens with no pretraining: expect thin world knowledge, factual errors and invented details.
• Language. Training data is mostly English.
• Hardware. A CUDA GPU is required.

Weights are Apache-2.0. Each training dataset keeps its own license; some carry attribution or non-commercial terms, so check them before commercial use.

---

Model family

Model	Notes	Hub
Senryu-2-1B	Senryu-2 architecture	this repo
Senryu	Previous generation (803M)	kerzgrr/Senryu
Haiku	Earlier generation (~655M)	kerzgrr/Haiku

---

References

• Gated DeltaNet-2: https://arxiv.org/abs/2605.22791
• Parallax attention: https://arxiv.org/abs/2605.29157
• Engram conditional memory: https://arxiv.org/abs/2601.07372
• flash-linear-attention, pinned at cbb0a72

@misc{senryu2_1b_2026,
  title={Senryu-2-1B: A 1.1B Hybrid Gated DeltaNet-2 + Parallax Chat Model with Hashed N-gram Memory},
  author={KerzgrAI},
  year={2026},
  url={https://huggingface.co/KerzgrAI/Senryu-2-1B}
}