Senryu-2.1-1B

Hybrid Gated DeltaNet-2 + Parallax chat model (1.09B) with 2.17B parameters of n-gram memory

By KerzgrAI: Senryu-2-1B trained on 2.06B more tokens, ending on a mix that is half new reasoning, system-prompt and tool data

What's new in 2.1

Senryu-2.1-1B is the second model on the Senryu-2 architecture. It continues Senryu-2-1B with the same architecture, tokenizer and chat template, so it is a drop-in replacement.

Improved reasoning. Half of its last 827M training tokens are new conversations with a full reasoning trace in each turn's <think> block.
Much better system prompt following. 119,137 of those conversations were answered under their own system prompts (personas, output formats, rules and constraints from Multifaceted-Collection, SystemChat 2.0 and SystemChat 1.1), and the earlier steps added system-prompt datasets such as SystemCheck.
A default system prompt and default tools. 36,929 conversations trained it under one Senryu system prompt with three tools, web-search, fetch-page and calculator, and rules to call them only when needed. inference.py and the demo start every chat with both.

Default system prompt

Conversations without a system prompt of their own were trained under this one, dated the day they were written. inference.py and the demo fill in today's date in the same format, such as Monday, October 26, 2026:

You are Senryu-2.1, a large language model from KerzgrAI. Today's date is {Weekday, Month D, YYYY}.

Tool Rules:
- Answer from your own knowledge whenever you can. Most requests need no tools.
- **Use web search sparingly, only when it is absolutely necessary:** for news and events after your training, live data such as prices, scores or weather, or a precise fact the user needs and you don't know. Never search for things you can answer yourself, such as explanations, definitions, how-tos, code, math, writing or advice, and never search just to double-check what you already know.
- When you must search, use one focused query and answer from its results. Search again only if they leave the question unanswered.
- Fetch a page only when the user links one or a search result you need is cut short.
- Use the calculator for arithmetic you can't reliably do in your head.

Formatting Rules:
- Use Markdown for lists, tables, and styling.
- Use ```code fences``` for all code blocks.
- Format file names, paths, and function names with `inline code` backticks.
- **For all mathematical expressions, you must use dollar-sign delimiters. Use $...$ for inline math and $$...$$ for block math. Do not use \(...\) or \[...\] delimiters.**
- For responses with many sections where some are more important than others, use collapsible sections (HTML details/summary tags) to highlight key information while allowing users to expand less critical details.

In Python, tiny_gdn.senryu_chat.senryu_system_prompt() returns it dated today. Your own system prompt works too, since most of the new conversations were answered under their own; pass --system "" or clear the demo's box to chat without any.

Default tools

Every conversation under the default prompt offered these three tools, in this order. Calls were rare: about 4% of those conversations call a tool at all.

Tool	Arguments	Result, as trained
`web-search`	`query` (string, required)	Tavily search JSON: `query`, `answer`, and `results` with each hit's `url`, `title` and `content`
`fetch-page`	`url` (string, required), `offset` (integer)	`URL:` and `Title:` lines, then the page or PDF as Markdown, 12,000 characters at a time, ending with the `offset` to read on
`calculator`	`expression` (string, required)	The value, for example `452093301`

The schemas, one per line as the system turn carries them:

{"type": "function", "function": {"name": "web-search", "description": "Search the web for a query.", "parameters": {"type": "object", "properties": {"query": {"type": "string", "description": "Search query."}}, "required": ["query"]}}}
{"type": "function", "function": {"name": "fetch-page", "description": "Fetch a web page or PDF by URL and return its readable text as Markdown.", "parameters": {"type": "object", "properties": {"url": {"type": "string", "description": "The http or https URL to fetch."}, "offset": {"type": "integer", "description": "Character offset to start reading from, for pages longer than one result."}}, "required": ["url"]}}}
{"type": "function", "function": {"name": "calculator", "description": "Evaluate a math expression and return the result.", "parameters": {"type": "object", "properties": {"expression": {"type": "string", "description": "A math expression in standard calculator notation, for example (3 + 4)^2 * sqrt(2) / ln(10). Use + - * / ^ (power), % (remainder), ! (factorial) and parentheses; the constants pi and e; and functions such as sqrt, cbrt, exp, ln, log (natural, or log(x, base)), log10, log2, sin, cos, tan, asin, acos, atan (radians), abs, round, floor, ceil, factorial, comb, perm, gcd, lcm, min, max, sum, mean and median."}}, "required": ["expression"]}}}

They are tiny_gdn.senryu_chat.SENRYU_TOOLS. Other Hermes-style tools work the same way: list them in the <tools> block (--tools-json in the CLI), and run each call yourself.

Prompt format

A tool round exactly as training laid it out, and as tiny_gdn.chatml.encode_chatml_messages builds it (the system prompt is shortened here):

<|begin_of_text|><|im_start|>system
You are Senryu-2.1, a large language model from KerzgrAI. Today's date is Monday, October 26, 2026.

Tool Rules:
…
- For responses with many sections where some are more important than others, use collapsible sections (HTML details/summary tags) to highlight key information while allowing users to expand less critical details.

You may call one or more functions to assist with the user query.
You are provided with function signatures within <tools></tools> XML tags:
<tools>
{"type": "function", "function": {"name": "web-search", …}}
{"type": "function", "function": {"name": "fetch-page", …}}
{"type": "function", "function": {"name": "calculator", …}}
</tools>

For each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:
<tool_call>
{"name": <function-name>, "arguments": <args-json-object>}
</tool_call><|im_end|>
<|im_start|>user
What is 48213 * 9377?<|im_end|>
<|im_start|>assistant
<|think|>
<think>
That's a five-digit product; the calculator is safer than mental arithmetic.
</think>

<tool_call>
{"name": "calculator", "arguments": {"expression": "48213 * 9377"}}
</tool_call><|im_end|>
<|im_start|>user
<|tool_response|>
452093301<|im_end|>
<|im_start|>assistant
<|think|>

Thinking. Each assistant turn starts with a control token: <|think|>\n, then a <think>…</think> block and the answer, or <|no_think|>\n and the answer only.
Tools. The <tools> block follows the system prompt after a blank line. With no tools, the system turn is the prompt alone.
Calls. The model writes any text first, then one <tool_call> block per call, each a JSON object with name and arguments. Independent calls share one turn.
Results. All results of a turn go back in one user turn, in call order, each starting with <|tool_response|>\n and separated by a newline. The model then answers in a new assistant turn, or calls again. With tiny_gdn, send each result as a {"role": "tool", "content": …} message and the encoder lays them out this way.

Install & run

pip install torch safetensors tokenizers huggingface_hub
hf download KerzgrAI/Senryu-2.1-1B inference.py --local-dir .
python inference.py

python inference.py --prompt "Explain recursion in two sentences."
python inference.py --prompt "What is 48213 * 9377?"
python inference.py --tools none --system ""
python inference.py --no-think --prompt "Reply in one sentence."
python inference.py --ngram-device cpu

inference.py downloads the weights, n-gram tables, tokenizer and tiny_gdn/, installs flash-linear-attention from git at the pinned commit (so Git must be on PATH) and installs the page readers fetch-page needs. A CUDA GPU is required: Gated DeltaNet-2 and Parallax run on Triton kernels.

The three default tools run automatically. web-search uses Tavily when TAVILY_API_KEY is set, and DuckDuckGo with Wikipedia otherwise, in the same JSON shape. fetch-page reads public http/https pages and PDFs only.

Flag	Default	Description
`--prompt`	—	One-shot user message
`--system`	the Senryu-2.1 prompt, dated today	System prompt, used verbatim; `""` for none
`--think` / `--no-think`	think	Assistant control prefix
`--tools`	`web-search,fetch-page,calculator`	Built-ins, comma-separated: `web-search`, `fetch-page`, `calculator`, `python`; `none` for no tools
`--tools-json`	—	Extra tool schemas (JSON list) for the `<tools>` block
`--max-tool-rounds`	`8`	Tool rounds per reply
`--temperature`	`0.6`	Sampling temperature (0.3 suits tool-calling turns)
`--top-p`	`0.95`	Nucleus sampling
`--top-k`	`50`	Top-k sampling
`--repetition-penalty`	`1.08`	On content tokens seen twice in the last `--repetition-window` (256) tokens
`--dry-multiplier`	`0.8`	DRY penalty on verbatim loops and re-pasted replies; `0` disables
`--dry-base`	`1.75`	How fast the DRY penalty grows per repeated token
`--dry-allowed-length`	`2`	Tokens a phrase may repeat before DRY starts
`--max-new-tokens`	`4096`	Max generation length
`--context-length`	`32768`	Conversation tokens kept (oldest dropped first)
`--device`	`cuda`	CUDA device, e.g. `cuda:1`
`--ngram-device`	same as `--device`	Where the n-gram tables live; `cpu` saves 4.3 GB of VRAM

Sampling matches the KerzgrAI API: the repetition penalty leaves Markdown and punctuation alone, and DRY breaks verbatim loops but stays off inside code fences and tool calls.

Interactive commands: /think /no_think /system … /reset /exit. Ctrl+C stops the current reply.

Model architecture


Core parameters	1,094,961,660 (embeddings included)
N-gram memory	2,166,398,464 table parameters: 2- and 3-gram hashes, four hash heads per order
Layers	28: 7 × (3 Gated DeltaNet-2 + 1 Parallax attention)
Hidden	1,536
MLP	SiTU-GLU 3,584
Attention	Parallax: 12 query and 4 KV heads of 128, RoPE on half of each head (θ = 10⁶)
Linear	Gated DeltaNet-2, 12 heads × 128, with a short convolution
Residuals	Four-branch gated residual stream (rank-128 gate)
Memory read	Before block 1, with a context-aware gate per residual branch, as in Engram
Vocab	65,536 BPE, tied embeddings
Context	32,768

Training

Continued from Senryu-2-1B's final SFT checkpoint (EMA weights) for 3,920 steps of 16 sequences × 32,768 tokens: 2.06B tokens, 1.41B of them assistant targets, in about 50 hours on one RTX PRO 6000 Blackwell GPU. Muon v2 + AdamW at a constant 5×10⁻⁴ after a 50-step warmup, and sparse Adam at 2.5×10⁻³ for the n-gram tables.

Steps 1–2,230: Senryu-2-1B's SFT mixture, refreshed with more system-prompt, chat and agentic data.
Steps 2,230–3,920: half new data, half replay of that mixture. The final mix, 289,976 conversations and 826.6M tokens, was trained once through:

Part	Conversations	Tokens	What it is
System prompts	119,137	225.0M	Multifaceted-Collection, SystemChat 2.0 and SystemChat 1.1 openings, answered with full reasoning under their own system prompts
Default prompt and tools	36,929	188.3M	smol-magpie-ultra openings, answered with full reasoning as Senryu-2.1 under the default system prompt and the three default tools; a quarter go on to a follow-up user turn
Replay	133,910	413.3M	The refreshed SFT mixture from steps 1–2,230, filtered of other models' self-descriptions

Only assistant turns are trained. The released EMA weights reach a loss of 1.433 on the final mix's held-out split, against 1.498 for the raw weights. sft_config.json holds the run's settings.

Files

File	Contents
`model.safetensors`	Core weights (EMA), bfloat16
`ngram_tables.safetensors`	N-gram tables (bfloat16), canonical ids and hash multipliers
`config.json`	Architecture
`tokenizer.json`, `special_token_ids.json`	Tokenizer and control-token ids
`inference.py`, `requirements.txt`, `tiny_gdn/`	Standalone chat CLI and the model code it imports
`sft_config.json`	Training settings
`windows_fla_patches/`	Import patches `inference.py` applies to fla on Windows

Limitations

Scale. 1.09B core parameters and no pretraining stage: expect thin world knowledge, factual errors and invented details.
Language. Training data is mostly English.
Hardware. A CUDA GPU is required.

Weights are Apache-2.0. Each training dataset keeps its own license; some carry attribution or non-commercial terms, so check them before commercial use.

Model family

Model	Notes	Hub
Senryu-2.1-1B	Senryu-2 architecture: better reasoning and system prompt following, default prompt and tools	this repo
Senryu-2-1B	First Senryu-2 model	`KerzgrAI/Senryu-2-1B`
Senryu	Previous generation (803M)	`kerzgrr/Senryu`
Haiku	Earlier generation (~655M)	`kerzgrr/Haiku`

References

Gated DeltaNet-2: https://arxiv.org/abs/2605.22791
Parallax attention: https://arxiv.org/abs/2605.29157
Engram conditional memory: https://arxiv.org/abs/2601.07372
flash-linear-attention, pinned at cbb0a72

@misc{senryu2_1_1b_2026,
  title={Senryu-2.1-1B: A 1.1B Hybrid Gated DeltaNet-2 + Parallax Chat Model with Hashed N-gram Memory},
  author={KerzgrAI},
  year={2026},
  url={https://huggingface.co/KerzgrAI/Senryu-2.1-1B}
}