RWKV-7 "Goose" with Expressive Dynamic State Evolution
Authors: name: Bo Peng, name: Ruichong Zhang, name: Daniel Goldstein, name: Eric Alcaide, name: Haowen Hou, name: Janna Lu, name: William Merrill, name: Guangyu Song, name: Kaifeng Tan, name: Saiteja Utpala, name: Nathan Wilce, name: Johan S. Wind, name: Tianyi Wu, name: Daniel Wuttke, name: Christian Zhou-Zheng

We present RWKV-7 "Goose", a new sequence modeling architecture, along with
pre-trained language models that establish a new state-of-the-art in downstream
performance at the 3 billion parameter scale on multilingual tasks, and match
current SoTA English language performance despite being trained on dramatically
fewer tokens than other top 3B models. Nevertheless, RWKV-7 models require only
constant memory usage and constant inference time per token. RWKV-7 introduces
a newly generalized formulation of the delta rule with vector-valued gating and
in-context learning rates, as well as a relaxed value replacement rule. We show
that RWKV-7 can perform state tracking and recognize all regular languages,
while retaining parallelizability of training. This exceeds the capabilities of
Transformers under standard complexity conjectures, which are limited to
TC^0. To demonstrate RWKV-7's language modeling capability, we also
present an extended open source 3.1 trillion token multilingual corpus, and
train four RWKV-7 models ranging from 0.19 billion to 2.9 billion parameters on
this dataset.
  To foster openness, reproduction, and adoption, we release our models and
dataset component listing at https://huggingface.co/RWKV, and our training and
inference code at https://github.com/RWKV/RWKV-LM all under the Apache 2.0
License.