RWKV-7 "Goose" with Expressive Dynamic State Evolution
Authors: name: Bo Peng; name: Ruichong Zhang; name: Daniel Goldstein; name: Eric Alcaide; name: Haowen Hou; name: Janna Lu; name: William Merrill; name: Guangyu Song; name: Kaifeng Tan; name: Saiteja Utpala; name: Nathan Wilce; name: Johan S. Wind; name: Tianyi Wu; name: Daniel Wuttke; name: Christian Zhou-Zheng
Published: 2025-03-18 17:31:05

Abstract: We present RWKV-7 "Goose", a new sequence modeling architecture, along with
pre-trained language models that establish a new state-of-the-art in downstream
performance at the 3 billion parameter scale on multilingual tasks, and match
current SoTA English language performance despite being trained on dramatically
fewer tokens than other top 3B models. Nevertheless, RWKV-7 models require only
constant memory usage and constant inference time per token. RWKV-7 introduces
a newly generalized formulation of the delta rule with vector-valued gating and
in-context learning rates, as well as a relaxed value replacement rule. We show
that RWKV-7 can perform state tracking and recognize all regular languages,
while retaining parallelizability of training. This exceeds the capabilities of
Transformers under standard complexity conjectures, which are limited to
TC^0. To demonstrate RWKV-7's language modeling capability, we also
present an extended open source 3.1 trillion token multilingual corpus, and
train four RWKV-7 models ranging from 0.19 billion to 2.9 billion parameters on
this dataset.
  To foster openness, reproduction, and adoption, we release our models and
dataset component listing at https://huggingface.co/RWKV, and our training and
inference code at https://github.com/RWKV/RWKV-LM all under the Apache 2.0
License.