rwkv-hils-3-m10

A 44.6M-parameter research language model. It interleaves RWKV-7 selective state-space layers (linear-time recurrence; Peng et al., RWKV-7) with HiLS layers — Hierarchical Landmark Sparse attention, which scores chunk summaries with an entropy-calibrated, low-rank-calibrated query, attends the Top-K chunks in full, and always attends a local window (Hu et al., Hierarchical Sparse Attention Done Right). Text-only, byte-level BPE, 4096-token vocabulary, 8192-token position budget.

Deviation from that paper: chunk summaries here use a shared learnable query instead of per-chunk landmark tokens (the paper's "w/o lmk, shared q_c" variant), because landmark tokens need extra tokens interleaved through every layer and that interacts badly with the recurrent blocks in this hybrid.

On the name. m10 is a tier label from this project's own ladder, keyed to depth and width rather than to parameters — this m10 is 44.6M parameters, not 10M. The ladder exists because the original tier labels were optimistic about parameter counts; the dimensions below are what is authoritative.

This is a research artifact, not a product. It has no instruction tuning and no safety tuning, and a couple of paragraphs of freeform generation are enough to find its failure modes — they are listed below rather than left for you to discover.

Files

file	what it is
model.safetensors	fp32 weights, unmodified state_dict tensor names, 44.6M params; safetensors, so loading executes no code
config.json	every architecture field plus step, params, model_type
tokenizer.json	byte-level BPE, 4096 vocab (trained for this project)
web/manifest.json, web/weights.bin, web/tokenizer.json	fp16 export for the in-browser runtime, half the download
cases.json	prompts and expected answers used by the browser demo
chat/	The chat tuned files
web/chat	The quantized fp16 weights for the chat version

Architecture

parameters	44,616,292 (fp32)
layers	12 = rwkv7, rwkv7, rwkv7, hils, hils, rwkv7, rwkv7, rwkv7, rwkv7, rwkv7, hils, hils
d_model / heads / d_head	512 / 8 / 64
feed-forward	4×
HiLS chunk / local window / Top-K	64 / 512 / 16 chunks
query-calibration rank	64
hils_share	full — every HiLS layer computes its own K/V and its own Top-K
max positions	8192
vocab	4096; ids 0–255 are reserved bracket symbols (see below), 256 PAD, 257 SEP, 258+ text

Two design details are worth knowing before you read the code:

• The vocabulary carries a synthetic task. Ids 0–255 are opening and closing bracket types for a Dyck-language task that shares the sequence with text, which is why a 4096-token text vocabulary sits inside a 4096 total. The model was trained on interleaved bracket and text windows.
• HiLS layers are not attention over the raw sequence. They attend over chunk summaries and a local window, which is what keeps the cost near-linear while still allowing exact retrieval inside the window. q_cal_rank is the low-rank calibration applied to the query before scoring summaries.

Measured behaviour

All numbers below are from the released checkpoint at step 10,000, measured locally (Arc A770, fp32) with the scripts in the repository.

metric	value	how it was measured
text loss	2.307	8 windows of 512–1024 tokens, teacher-forced CE, from a differently-seeded slice of the same corpora — see the caveat below
Dyck, depth ≥ 10, close-step accuracy	0.949	next-symbol accuracy where the target is a closing bracket, which is the position that requires the stack; overall accuracy is ~0.5 by construction, since openings are uniform random
needle-in-a-haystack, single needle	1.00 @ 1K, 1.00 @ 4K	teacher-forced exact match over the answer span, 4 seeds
needle-in-a-haystack, multi-needle	0.00	same, but this variant was never generated during training
needle-in-a-haystack, variable tracking	0.00	same, never generated during training
needle-in-a-haystack, real-prose families	0.00	same; these paraphrase-based families postdate this checkpoint

The last three rows are the honest headline: retrieval in this model is specific to the retrieval task it was trained on. It solves the single-needle variant perfectly and does nothing at all on variants that never appeared in its data. That is a statement about the training data, not about the architecture — a newer, deliberately broader needle generator exists in the repository and is not what this checkpoint was trained with.

Known failure modes

• Repetition. Long freeform generations collapse into repeating phrases or into short list-like lines. Sampling-side penalties (DRY, a targeted continuation penalty, a frequency penalty) were implemented and measured: they move which attractor it lands in, not whether it lands in one. At this scale it is a property of the model, not of the sampler.
• No instruction following. Base model. Prompts are continuations, not requests.
• Short effective range for prose. Retrieval holds at 4K for the trained variant; general coherence over long documents was not measured here and should not be assumed.
• Incorrect/conflicting chat tokens The chat model was finetuned using tokens that were already present in its output due to an oversight in planning.
• Chat coherence is very weak The chat model is able to sometimes stay on topic, but factuality or general coherence is not effective.

Running it

Python — chat.py at the root of this repository is the entry point (interactive REPL and one-shot mode):

uv run chat.py Vibelet/rwkv-hils-3-m10 --prompt "The sea was" --max_new 200
uv run chat.py /path/to/local/dir --device cpu     # or xpu / cuda

# or install it and use the console script
uv pip install . && rwkv-hils-3 Vibelet/rwkv-hils-3-m10 --prompt "The sea was"

A repository id, a local directory holding model.safetensors + config.json, a .pt training checkpoint and a directory of checkpoints are all accepted. Runs on CUDA, Intel XPU and CPU; fp32 inference, ~180 MB resident. The fast kernels need triton, but a host without it still works: the model falls back to the reference recurrence, which produces identical tokens (verified by running with the triton import blocked) and only changes the speed.

Browser — the demo in the repository runs the model entirely client-side, with WebGPU when available and a CPU fallback:

uv run demo/web/serve.py     # serves with no-store, port 8123
# open http://127.0.0.1:8123/demo/web/index.html

?backend=cpu forces the reference path. The page can also point at any hosted copy of the weights with ?model=<base-url>, which is how the public demo site loads the fp16 export from this repository.

From the Hub directly — download the files and use web/ as the base URL; the browser runtime fetches one manifest and one flat blob, and views every tensor in place rather than copying them.

The fp16 export was checked against the fp32 weights in the browser runtime before shipping: max |Δlogit| 9.1e-3 over 24 decode steps, with the greedy token sequence identical. That is a constant quantisation offset and not a backend disagreement, so the demo's CPU-versus-WebGPU comparison still holds within either export.

Training

steps	10,000
wall clock	42 hours
precision	fp32
hardware	a single NVIDIA L40(The intial 2000 steps were performed on an Arc A770)
learning-rate multiplier	halved once mid-run, automatically, when the needle-in-a-haystack score for the trained variant dropped by more than 0.05
objective	cross-entropy with per-token weights: bracket and text windows at 1.0, needle, question and answer tokens at 5.0

Training mixes three kinds of window: synthetic bracket sequences (curriculum over depth), real text, and needle-in-a-haystack examples rendered as text. The mix shifts over the run — bracket-heavy early, text-dominant later, with the needle task present from the point it becomes measurable.

Data. Public corpora, drawn through a cached token pipeline:

corpus	size in the local cache	provenance
Wikipedia (enwiki, 20231101 dump)	238M tokens	Wikimedia, CC BY-SA
Project Gutenberg English	158M tokens	public domain; filtered to Library-of-Congress class P (language and literature)
fineweb-edu	145M tokens	ODC-By, CommonCrawl-derived
literature	8.1M tokens	ACOSharma/literature
fiction	2.2M tokens	public-domain novels, one document per book

Each listed source was drawn to between 82.5% and 86.7% of its target epochs by final step, so the run consumed these corpora at close to one pass. The pipeline also renders synthetic bracket and retrieval windows. This release ships inference and the browser demo only; the training pipeline is not part of it, and the dimensions, step count, objective and wall clock above are what the released weights were produced with.

Caveat on the text-loss figure. Because the corpora were consumed at close to one epoch, the validation windows — drawn with a different seed from the same sources — are not strictly held out. At 44.6M parameters and under one epoch of exposure, memorisation of any specific window is unlikely, but the number should be read as an in-corpus loss rather than a generalisation measurement. A loss on text the run never saw is not reported here.

Content note: the corpus above is public web text and public-domain literary fiction. The model can produce text that is unsuitable for some audiences and has no filtering of any kind.

Chat conditioning

steps	1,000
wall clock	9 hours
precision	fp32
hardware	a single NVIDIA L40
learning-rate multiplier	2.5e-5
objective	cross-entropy with masked per-token weights

Data

The single data source for the fine tuning is the wildchat dataset and drew upon 260 million tokens from the dataset. Toxic discussions and non-english language text were filtered out.

Limitations and intended use

Intended for experimentation with hybrid recurrent/attention architectures, sparse-attention retrieval behaviour, and small-model training dynamics. It is not suitable for production, for factual applications, or for any use where outputs need to be correct, safe or attributable. Do not treat its outputs as information about the world.

The chat version is very confident but also almost always very wrong.

Licence

MIT. The training corpora keep their own licences, listed above.