RWKV-WORLD-4
Author: CosmoLM
Pull request: no
Labels: name: New model

### Model description

BlinkDL/rwkv-4-world is a repo present on Huggingface i want the model's tokenizer and the model to be added to the Transformers Lib. 

### Open source status

- [X] The model implementation is available
- [X] The model weights are available

### Provide useful links for the implementation

_No response_

Comments:
Hi @CosmoLM, thanks for opening this model request! 

The RWKV-4 model already exists in transformers -- [PR](https://github.com/huggingface/transformers/pull/22797), [docs](https://huggingface.co/docs/transformers/v4.31.0/en/model_doc/rwkv#rwkv-attention-and-the-recurrent-formulas). To enable loading the model through `Rwkv.from_pretrained`, the checkpoints would need to be converted and model configs push to the hub using [the conversion script.](https://github.com/huggingface/transformers/blob/8021c684ec3023295513be36bdc30e27e6f28cfc/src/transformers/models/rwkv/convert_rwkv_checkpoint_to_hf.py#L4)  

I'd suggest opening a discussion on the hub to see if the repo owners would be interested in doing this. 
 

The RWKV-pile models are available but not the RWKV-world models because
its tokenizer is not in the json format it is in txt format.

On Wed, 2 Aug, 2023, 4:24 pm amyeroberts, ***@***.***> wrote:

> Hi @CosmoLM <https://github.com/CosmoLM>, thanks for opening this model
> request!
>
> The RWKV-4 model already exists in transformers -- PR
> <https://github.com/huggingface/transformers/pull/22797>, docs
> <https://huggingface.co/docs/transformers/v4.31.0/en/model_doc/rwkv#rwkv-attention-and-the-recurrent-formulas>.
> To enable loading the model through Rwkv.from_pretrained, the checkpoints
> would need to be converted and model configs push to the hub using the
> conversion script.
> <https://github.com/huggingface/transformers/blob/8021c684ec3023295513be36bdc30e27e6f28cfc/src/transformers/models/rwkv/convert_rwkv_checkpoint_to_hf.py#L4>
>
> I'd suggest opening a discussion on the hub to see if the repo owners
> would be interested in doing this.
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/transformers/issues/25253#issuecomment-1661993346>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/BA7FALGYW7ERQ3LODEA6NADXTIWVPANCNFSM6AAAAAA3A3B6CY>
> .
> You are receiving this because you were mentioned.Message ID:
> ***@***.***>
>


Is there an ETA for this?

I've just made a quick hack to construct RWKV-world tokenizer (fast), using GPT2TokenizerFast with WordPiece instead of BPE:

```
#! /usr/bin/python3
import json
from ast import literal_eval
from urllib.request import urlopen
from transformers import GPT2TokenizerFast
from transformers.models.gpt2.tokenization_gpt2 import bytes_to_unicode
from tokenizers.models import WordPiece
with urlopen("https://github.com/BlinkDL/ChatRWKV/raw/refs/heads/main/tokenizer/rwkv_vocab_v20230424.txt") as r:
  v=[s.split(" ") for s in r.read().decode("utf-8").split("\n") if s>""]
d=literal_eval("{"+",".join(t[0]+":"+" ".join(t[1:-1]) for t in v)+"}")
d[0]="
"
v,x={},bytes_to_unicode()
for i in range(len(d)):
  k="".join(x[j] for j in (d[i].encode("utf-8") if type(d[i])==str else d[i]))
  v[k]=i
m=["#version: 0.2"]
for t in v:
  if v[t]>0:
    for j in reversed(range(1,len(t))):
      if t[:j] in v and t[j:] in v:
        m.append(t[:j]+" "+t[j:])
with open("vocab.json","w",encoding="utf-8") as w:
  json.dump(v,w,indent=2,ensure_ascii=False)
with open("merges.txt","w",encoding="utf-8") as w:
  print("\n".join(m),file=w)
tokenizer=GPT2TokenizerFast("vocab.json","merges.txt")
w=WordPiece(v)
w.unk_token=d[0]
w.continuing_subword_prefix=""
tokenizer.backend_tokenizer.model=w
tokenizer.__class__.__name__="PreTrainedTokenizerFast"
tokenizer.save_pretrained(".")
```

where I added 0th token `
` since [rwkv_vocab_v20230424.txt](https://github.com/BlinkDL/ChatRWKV/blob/main/tokenizer/rwkv_vocab_v20230424.txt) lacks 0th token. Well, how do we write RwkvTokenizer and RwkvTokenizerFast?