RWKV7-2.9B-20260805

RWKV-7 “Goose” · constant-state recurrent language modeling

---

Model introduction

This is an official BlinkDL release of RWKV-7 Goose in Hugging Face Transformers format. RWKV-7 is an attention-free recurrent architecture with a constant-size recurrent state and constant inference work per generated token. Training remains parallelizable.

This checkpoint is a base model pretrained with web, code, synthetic, instruction, chat, and reasoning data. It is suitable for evaluation, post-training, and fine-tuning; the included chat template is a prompt interface, not a claim that the checkpoint is a safety-aligned assistant.

The Transformers integration, conversion, release packaging, Fast Tokenizer, and optional TileLang inference implementation are distributed with this release.

Highlights

• Constant recurrent state: memory does not grow like an attention KV cache.
• Native Transformers layout: standard config, sharded safetensors, generation, recurrent cache continuation, training, and LoRA workflows.
• Exact Fast Tokenizer: self-contained Rust-backed tokenizer.json, generated from the canonical RWKV World byte vocabulary during conversion.
• Chat-ready: chat_template.jinja supports system, multi-turn, thinking, and strict model-generated tool-call prompts.
• Optional optimized runtime: the isolated inference/ bundle provides PyTorch fallback and TileLang acceleration without changing the standard model root.

Model overview

Field	Value
Repository	RWKV/RWKV7-2.9B-20260805
Architecture class	Rwkv7ForCausalLM
Public size label	2.9B
Source parameters	2,948,065,280
Serialized parameters	2,948,065,280
Synthesized compatibility tensors	0
Layers	32
Hidden / FFN size	2560 / 10240
Heads / head size	40 / 64
Vocabulary	65536
Training context	16384 tokens
Weight dtype	bfloat16
Numerical conversion	source dtype preserved
Metadata profile	g1i
Metadata provenance	locked-profile
Source checkpoint	BlinkDL/rwkv7-g1/rwkv7-g1i-2.9b-20260805-ctx16384.pth
Source SHA-256	ac1ae23d0e65c1d35ba523eacd81a2a4dacb7b886479909bbff34f312e766320

Transformers quickstart

Native rwkv7 auto-class registration requires Transformers 5.15 or a current source checkout until that release is available.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "RWKV/RWKV7-2.9B-20260805"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, dtype=torch.bfloat16)

The recurrent cache returned by the model can be passed back for incremental decoding. Use an attention_mask for padded batches.

Chat quickstart

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "RWKV/RWKV7-2.9B-20260805"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    dtype=torch.bfloat16,
).to("cuda")

messages = [{"role": "user", "content": "Explain why RWKV uses constant state."}]
input_ids = tokenizer.apply_chat_template(
    messages,
    tokenize=True,
    add_generation_prompt=True,
    thinking=False,
    return_tensors="pt",
).to(model.device)

output = model.generate(
    input_ids,
    max_new_tokens=256,
    do_sample=True,
    temperature=1.0,
    top_p=0.5,
    eos_token_id=0,
    pad_token_id=0,
)
print(tokenizer.decode(output[0, input_ids.shape[1]:], skip_special_tokens=True))

Set thinking=True for the RWKV thinking prefix. The intentional generation prefixes are Assistant: <think></think and Assistant: <think; do not append a closing > to them. Reference stops are token ID 0 and \n\nUser:.

Strip trailing spaces from user input. The official RWKV prompt guide is available in RWKV7-G1x-templates.txt.

Optimized local inference

Install the versions listed in inference/requirements.txt, then run the bundled interactive chat:

python inference/generate.py --model RWKV/RWKV7-2.9B-20260805 --backend auto --interactive

Or independent prompts separated by blank lines:

python inference/generate.py \
  --model RWKV/RWKV7-2.9B-20260805 \
  --backend auto \
  --input-file prompts.txt

--backend auto uses validated exact optimized boundaries and otherwise falls back to PyTorch. Full explicit TileLang execution can change floating-point operation order and requires checkpoint-, dtype-, shape-, and device-specific parity validation.

Tokenizer

The model root contains one self-contained tokenizer artifact: tokenizer.json. Textual vocab.json and rwkv_vocab_v20230424.txt files are intentionally omitted because they would duplicate the tokenizer used by Transformers.

Intended use and limitations

• This is a base causal language model. Quality, instruction following, and language behavior depend on the checkpoint and downstream prompting or post-training.
• Assisted or speculative decoding that requires recurrent-cache rollback is not supported without retaining prior state snapshots.
• Optimized support depends on GPU architecture, dtype, batch, and shape. Unsupported auto configurations fall back to pure PyTorch.
• Explicit full TileLang execution can change floating-point operation order and requires checkpoint-, dtype-, shape-, and device-specific parity validation.
• No safety, bias, toxicity, factuality, or high-stakes-use evaluation is claimed by this model card.

License and provenance

The model weights use the locked profile license apache-2.0. The exported inference bundle is licensed separately under Apache-2.0. See NOTICE and the source checkpoint link above for provenance.

Citation

@misc{peng2025250314456,
  title         = {RWKV-7 "Goose" with Expressive Dynamic State Evolution},
  author        = {Bo Peng and Ruichong Zhang and Daniel Goldstein and Eric Alcaide and Xingjian Du and Haowen Hou and Jiaju Lin and Jiaxing Liu and Janna Lu and William Merrill and Guangyu Song and Kaifeng Tan and Saiteja Utpala and Nathan Wilce and Johan S. Wind and Tianyi Wu and Daniel Wuttke and Christian Zhou-Zheng},
  year          = {2025},
  eprint        = {2503.14456v2},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CL},
  url           = {https://arxiv.org/abs/2503.14456v2},
}