[RWKV] Add RWKV5 model and RWKVWorldTokenizer
Author: BBuf
Pull request: yes

Add RWKVWorldTokenizer for rwkv5 series model.

The tokenizer has been used in:

- [RWKV/rwkv-5-world-1b5](https://huggingface.co/RWKV/rwkv-5-world-1b5)
- [RWKV/rwkv-5-world-3b](https://huggingface.co/RWKV/rwkv-5-world-3b)
- [RWKV/rwkv-4-world-169m](https://huggingface.co/RWKV/rwkv-4-world-169m)
- [RWKV/rwkv-4-world-430m](https://huggingface.co/RWKV/rwkv-4-world-430m)
- [RWKV/rwkv-4-world-1b5](https://huggingface.co/RWKV/rwkv-4-world-1b5)
- [RWKV/rwkv-4-world-3b](https://huggingface.co/RWKV/rwkv-4-world-3b)
- [RWKV/rwkv-4-world-7b](https://huggingface.co/RWKV/rwkv-4-world-7b)

and lambda test in https://github.com/BBuf/RWKV-World-HF-Tokenizer/blob/main/check_lambda/lambda_hf.py

@xianbaoqian 

Comments:
Hey feel free to ping me when this is ready! 🤗 

Hi, pr ready now  🤗. @ArthurZucker 

Ok! Thanks, I'll review now, but will let @amyeroberts handle the rest as I'll be off for a week 😉 

> Thanks for the PR! Could you explain the motivation behind not using the fast tokenizer, and whether this tokenizer / slow implem of GPT2 for example.
> 
> Mostly, this should need a new folder as it's a new model ! If we use the GPT2Tokenizer implementation then we can also just add a .md file ( like we did for flan T5 for example)

The model implementation is the same, only the tokenizer has different options. The tokenizer implemented in this PR is for the RWKV4 World model, but the implementation of the RWKV4 World model is exactly the same as the existing RWKV model implementation.

@ArthurZucker Hello, I have implemented the RWKV5 model and the RWKVWorldTokenizer it requires. Please review again. Thank you.

> Thanks for the PR! Could you explain the motivation behind not using the fast tokenizer, and whether this tokenizer / slow implem of GPT2 for example.
> 
> Mostly, this should need a new folder as it's a new model ! If we use the GPT2Tokenizer implementation then we can also just add a .md file ( like we did for flan T5 for example)

@BBuf, thanks for helping push this!

Im from the RWKV team. So i can help explain this part.

The main motivation for the world tokenizer is to improve support for multi-lingual dataset, within the RWKV generations of models. Especially in character based languages, or languages without "spaces". This benefit applies to for european or nordic languages.

Okay, understood! So this new model uses a word level tokenizer, which can be supported both in transformers (by adding a new tokenizer, with a simple vocab / the code you are proposing) and `tokenizers` which natively has a [`WordLevel`](https://huggingface.co/docs/tokenizers/api/tokenizer#tokenizers.models.WordLevel) tokenizer ! 

I'm thrilled to help you get this merged 😉 

> Okay, understood! So this new model uses a word level tokenizer, which can be supported both in transformers (by adding a new tokenizer, with a simple vocab / the code you are proposing) and `tokenizers` which natively has a [`WordLevel`](https://huggingface.co/docs/tokenizers/api/tokenizer#tokenizers.models.WordLevel) tokenizer !
> 
> I'm thrilled to help you get this merged 😉

Hello, could you take another look at this PR? The recent few commits have added support for batch inference, and I feel it's getting close to being merged.

Sure I’ll review today! 🤗

> Okay, understood! So this new model uses a word level tokenizer, which can be supported both in transformers (by adding a new tokenizer, with a simple vocab / the code you are proposing) and `tokenizers` which natively has a [`WordLevel`](https://huggingface.co/docs/tokenizers/api/tokenizer#tokenizers.models.WordLevel) tokenizer !
> 
> I'm thrilled to help you get this merged 😉

I would not call it a "word level" more of a "trie tokenizer", spaces are just simply another character with no special meaning - If that makes sense.

But yes, that in concept this tokenizer could be used for non RWKV architecture, and there is nothing stopping anyone from using our older GPT-neox tokenizer on our newer architecture.

Do let me know if I can clarify anything else from our end, or help in this merge =)

Okay! I'll let you know, sorry I got caught up in sprints here and there but will review this early next week 🤗 

All the progress look good! Ping me whenever for another review! 🤗 

Now that rwkv5 pretrained model is out, will this get merged?

I'll review again and help merge it asap! 

This yields the following:
```python 
>>> from transformers import Rwkv5Tokenizer
>>> tokenizer = Rwkv5Tokenizer("/Users/arthurzucker/Work/transformers/rwkv.txt")
>>> prompt = "Hey how are you? 男：听说你们公司要派你去南方工作"
>>> ids = tokenizer.encode(prompt)

>>> print(ids)
[0, 6037, 21887, 21338, 22851, 64, 65517, 14631, 19181, 11095, 16765, 10494, 10432, 10708, 11059, 16533, 13848, 10494, 11015, 10964, 13066, 12167, 10490]
>>> print(tokenizer.tokenize(prompt))
['Hey', ' how', ' are', ' you', '?', ' ', '男', '：', '听', '说', '你', '们', '公', '司', '要', '派', '你', '去', '南', '方', '工', '作']
>>> print(tokenizer.convert_ids_to_tokens(tokenizer.encode(prompt)))
['
', 'Hey', ' how', ' are', ' you', '?', ' ', '男', '：', '听', '说', '你', '们', '公', '司', '要', '派', '你', '去', '南', '方', '工', '作']
>>> print(tokenizer.convert_tokens_to_string(tokenizer.convert_ids_to_tokens(tokenizer.encode(prompt))))

Hey how are you? 男：听说你们公司要派你去南方工作
>>> print(tokenizer.decode(tokenizer.encode(prompt)))

Hey how are you? 男：听说你们公司要派你去南方工作
```

> This yields the following:
> 
> ```python
> >>> from transformers import Rwkv5Tokenizer
> >>> tokenizer = Rwkv5Tokenizer("/Users/arthurzucker/Work/transformers/rwkv.txt")
> >>> prompt = "Hey how are you? 男：听说你们公司要派你去南方工作"
> >>> ids = tokenizer.encode(prompt)
> 
> >>> print(ids)
> [0, 6037, 21887, 21338, 22851, 64, 65517, 14631, 19181, 11095, 16765, 10494, 10432, 10708, 11059, 16533, 13848, 10494, 11015, 10964, 13066, 12167, 10490]
> >>> print(tokenizer.tokenize(prompt))
> ['Hey', ' how', ' are', ' you', '?', ' ', '男', '：', '听', '说', '你', '们', '公', '司', '要', '派', '你', '去', '南', '方', '工', '作']
> >>> print(tokenizer.convert_ids_to_tokens(tokenizer.encode(prompt)))
> ['
', 'Hey', ' how', ' are', ' you', '?', ' ', '男', '：', '听', '说', '你', '们', '公', '司', '要', '派', '你', '去', '南', '方', '工', '作']
> >>> print(tokenizer.convert_tokens_to_string(tokenizer.convert_ids_to_tokens(tokenizer.encode(prompt))))
> 
Hey how are you? 男：听说你们公司要派你去南方工作
> >>> print(tokenizer.decode(tokenizer.encode(prompt)))
> 
Hey how are you? 男：听说你们公司要派你去南方工作
> ```

Thank you for your advice. The main problem here is that the original tokenizer implementation(https://github.com/BlinkDL/ChatRWKV/tree/main/tokenizer) does not have bos, eos, or pad token, but bos_token_id, eos_token_id, and pad_token_id are all set to `0` . In my implementation on Hugging Face, I have simulated this situation. But now I am unsure what to set for bos, eos, and pad token, as it seems that setting any token would not meet expectations. Therefore, it feels like this tokenizer is a special hack case. I would like to ask if it is acceptable for the tokenizer's definition not to be merged into Transformers repository, and only merge the implementation of the RWKV5 model. Then, this custom tokenizer implementation could be placed in the corresponding repository on Hugging Face, for example: https://huggingface.co/RWKV/HF_v5-Eagle-7B . 

In the code I provided I manually set `self._added_tokens_decoder = {0:AddedToken(bos_token)}` 
which forces the token 0 to be the `bos_token`. We can of course force any other behaviour that way, but we only need to define a string token. This will be *very* useful to the community as a whole for any SFT training, as it's the expected api. 

Whether or not the original tokenizer has a `token`, it has `token_id=0` which means we can choose the content of the token. I used `
` but we should use something like `
` just to make sure it doesn't exist. This should solve all the issues you are having . Doing something like `tokenizer.encode("
")` will yield `0`. which is what we want 

WDYT? 

In general the model was not trained with a special bos_token, or special pad_token in mind (we use 0, and masking for padding)

So for all these tokens, we typically just use token 0 as fallback, and it "generally works" if that makes sense - so i think defaulting for all these tokens as 0 makes sense to me (coming from the trainer / model team side)

I ran into a bug with this tokenizer, [sourced from here](https://huggingface.co/RWKV/rwkv-5-world-1b5/blob/main/modeling_rwkv5.py). I'm not sure how much the two code bases have diverged at this point, but @BlinkDL asked me to report it in this PR.

My issue relates to this code:
```py
tokenized_batches = []
with tqdm(total=len(batches)) as pbar:
    for batch in batches:
        tokenized = tokenizer(
            batch,
            max_length=block_size,
            stride=stride,
            padding="max_length",
            return_overflowing_tokens=True,
            truncation=True,
            return_tensors="np",
        )
        tokenized_batches.append(tokenized["input_ids"])
        pbar.update(1)

tokens = np.concatenate(tokenized_batches)
```
Typically, the tokenizer should pad these batches to a consistent length, but that's not happening here:
```
  File "/usr/local/lib/python3.10/dist-packages/aigen/datasets.py", line 188, in encode_tokens
    tokens = np.concatenate(tokenized_batches)
ValueError: all the input array dimensions except for the concatenation axis must match exactly, but along dimension 1, the array at index 0 has size 24226 and the array at index 1 has size 26680
```
The fix for me was rather simple, but I feel like it should probably be handled by the tokenizer itself:
```
padded_batches = []
for batch in tokenized_batches:
    # Calculate the number of padding tokens needed
    padding_length = max_len - batch.shape[1]
    # Pad the batch and add it to the padded_batches list
    if padding_length > 0:
        padded_batch = np.pad(batch, ((0, 0), (0, padding_length)), mode='constant', constant_values=tokenizer.pad_token_id)
    else:
        padded_batch = batch
    padded_batches.append(padded_batch)
```
Just a PSA. Thanks!

Can you try with this one: https://github.com/huggingface/transformers/pull/26963#pullrequestreview-1861671950

> Can you try with this one: [#26963 (review)](https://github.com/huggingface/transformers/pull/26963#pullrequestreview-1861671950)

Thank you, I will have a try.

> This yields the following:
> 
> ```python
> >>> from transformers import Rwkv5Tokenizer
> >>> tokenizer = Rwkv5Tokenizer("/Users/arthurzucker/Work/transformers/rwkv.txt")
> >>> prompt = "Hey how are you? 男：听说你们公司要派你去南方工作"
> >>> ids = tokenizer.encode(prompt)
> 
> >>> print(ids)
> [0, 6037, 21887, 21338, 22851, 64, 65517, 14631, 19181, 11095, 16765, 10494, 10432, 10708, 11059, 16533, 13848, 10494, 11015, 10964, 13066, 12167, 10490]
> >>> print(tokenizer.tokenize(prompt))
> ['Hey', ' how', ' are', ' you', '?', ' ', '男', '：', '听', '说', '你', '们', '公', '司', '要', '派', '你', '去', '南', '方', '工', '作']
> >>> print(tokenizer.convert_ids_to_tokens(tokenizer.encode(prompt)))
> ['
', 'Hey', ' how', ' are', ' you', '?', ' ', '男', '：', '听', '说', '你', '们', '公', '司', '要', '派', '你', '去', '南', '方', '工', '作']
> >>> print(tokenizer.convert_tokens_to_string(tokenizer.convert_ids_to_tokens(tokenizer.encode(prompt))))
> 
Hey how are you? 男：听说你们公司要派你去南方工作
> >>> print(tokenizer.decode(tokenizer.encode(prompt)))
> 
Hey how are you? 男：听说你们公司要派你去南方工作
> ```



> This yields the following:
> 
> ```python
> >>> from transformers import Rwkv5Tokenizer
> >>> tokenizer = Rwkv5Tokenizer("/Users/arthurzucker/Work/transformers/rwkv.txt")
> >>> prompt = "Hey how are you? 男：听说你们公司要派你去南方工作"
> >>> ids = tokenizer.encode(prompt)
> 
> >>> print(ids)
> [0, 6037, 21887, 21338, 22851, 64, 65517, 14631, 19181, 11095, 16765, 10494, 10432, 10708, 11059, 16533, 13848, 10494, 11015, 10964, 13066, 12167, 10490]
> >>> print(tokenizer.tokenize(prompt))
> ['Hey', ' how', ' are', ' you', '?', ' ', '男', '：', '听', '说', '你', '们', '公', '司', '要', '派', '你', '去', '南', '方', '工', '作']
> >>> print(tokenizer.convert_ids_to_tokens(tokenizer.encode(prompt)))
> ['
', 'Hey', ' how', ' are', ' you', '?', ' ', '男', '：', '听', '说', '你', '们', '公', '司', '要', '派', '你', '去', '南', '方', '工', '作']
> >>> print(tokenizer.convert_tokens_to_string(tokenizer.convert_ids_to_tokens(tokenizer.encode(prompt))))
> 
Hey how are you? 男：听说你们公司要派你去南方工作
> >>> print(tokenizer.decode(tokenizer.encode(prompt)))
> 
Hey how are you? 男：听说你们公司要派你去南方工作
> ```

I tried this tokenizer, but it seems that I can't get the expected results.

![图片](https://github.com/huggingface/transformers/assets/35585791/8a767663-799d-4925-aef2-52920eb6117e)

I would like to ask if the `rwkv.txt` in the code you provided is the original file from https://github.com/BlinkDL/ChatRWKV/blob/main/tokenizer/rwkv_vocab_v20230424.txt.


I converted the vocab to the appropriate format to read it, sorry I forgot that step. Will push it now. I used your tokenizer's `encoder`

> I converted the vocab to the appropriate format to read it, sorry I forgot that step. Will push it now. I used your tokenizer's `encoder`

Okay, thanks.

I used this :
```python 
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("RWKV/rwkv-5-world-1b5", trust_remote_code=True)

with open("/Users/arthurzucker/Work/transformers/rwkv.txt", "wb") as f:
    for index, token in tokenizer.encoder.item():
        f.write(token +  b"\n")


tokenizer = Rwkv5Tokenizer("/Users/arthurzucker/Work/transformers/rwkv.txt")
prompt = "Hey how are you? 男：听说你们公司要派你去南方工作"
ids = tokenizer.encode(prompt)
print(ids)
print(tokenizer.tokenize(prompt))
print(tokenizer.convert_ids_to_tokens(tokenizer.encode(prompt)))
print(tokenizer.convert_tokens_to_string(tokenizer.convert_ids_to_tokens(tokenizer.encode(prompt))))
print(tokenizer.decode(tokenizer.encode(prompt)))
```

Pushed the tokenizer here: https://huggingface.co/ArthurZ/rwkv-5

> Pushed the tokenizer here: https://huggingface.co/ArthurZ/rwkv-5

Okay, got it.

> Pushed the tokenizer here: https://huggingface.co/ArthurZ/rwkv-5

Hello, I encountered an error while testing this tokenizer, but I'm not sure how to resolve it.

```
ERROR: test_added_token_serializable (tests.models.rwkv5.test_tokenization_rwkv5.RWKV5TokenizationTest.test_added_token_serializable) [Rwkv5Tokenizer]
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/bbuf/工作目录/RWKV/transformers/tests/test_tokenization_common.py", line 2204, in test_added_token_serializable
    tokenizer.from_pretrained(tmp_dir_name)
  File "/opt/homebrew/lib/python3.11/site-packages/transformers-4.38.0.dev0-py3.11.egg/transformers/tokenization_utils_base.py", line 2031, in from_pretrained
    return cls._from_pretrained(
           ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/transformers-4.38.0.dev0-py3.11.egg/transformers/tokenization_utils_base.py", line 2263, in _from_pretrained
    tokenizer = cls(*init_inputs, **init_kwargs)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/transformers-4.38.0.dev0-py3.11.egg/transformers/models/rwkv5/tokenization_rwkv5.py", line 133, in __init__
    self._added_tokens_decoder = {0:AddedToken(bos_token)}
                                    ^^^^^^^^^^^^^^^^^^^^^
TypeError: argument 'content': 'AddedToken' object cannot be converted to 'PyString'
```

Could you share your testing! It should be easy to solve: 
```diff 
-self._added_tokens_decoder = {0:AddedToken(bos_token)}
+self._added_tokens_decoder = {0:AddedToken(str(bos_token))}
```
should fix it 

> Could you share your testing! It should be easy to solve:
> 
> ```diff
> -self._added_tokens_decoder = {0:AddedToken(bos_token)}
> +self._added_tokens_decoder = {0:AddedToken(str(bos_token))}
> ```
> 
> should fix it

Thank you. My test is the `test_tokenization_rwkv5.py` in this PR. Currently, some tokenizer tests have not passed, and I am trying to solve them. 

```
....Saving vocabulary to /var/folders/k6/cvg5yzcs1957ywm2m80_y3dw0000gn/T/tmpqpufnio8/rwkv_vocab_v20230424.txt: vocabulary indices are not consecutive. Please check that the vocabulary is not corrupted!
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
.....Truncation was not explicitly activated but `max_length` is provided a specific value, please use `truncation=True` to explicitly truncate examples to max length. Defaulting to 'longest_first' truncation strategy. If you encode pairs of sequences (GLUE-style) with the tokenizer you can select this strategy more precisely by providing a specific strategy to `truncation`.
...s..Saving vocabulary to /var/folders/k6/cvg5yzcs1957ywm2m80_y3dw0000gn/T/tmp_epovmhw/rwkv_vocab_v20230424.txt: vocabulary indices are not consecutive. Please check that the vocabulary is not corrupted!
.....E......E....Token indices sequence length is longer than the specified maximum sequence length for this model (2108 > 100). Running this sequence through the model will result in indexing errors
Be aware, overflowing tokens are not returned for the setting you have chosen, i.e. sequence pairs with the 'longest_first' truncation strategy. So the returned list will always be empty even if some tokens have been removed.
Be aware, overflowing tokens are not returned for the setting you have chosen, i.e. sequence pairs with the 'longest_first' truncation strategy. So the returned list will always be empty even if some tokens have been removed.
Be aware, overflowing tokens are not returned for the setting you have chosen, i.e. sequence pairs with the 'longest_first' truncation strategy. So the returned list will always be empty even if some tokens have been removed.
Be aware, overflowing tokens are not returned for the setting you have chosen, i.e. sequence pairs with the 'longest_first' truncation strategy. So the returned list will always be empty even if some tokens have been removed.
.Token indices sequence length is longer than the specified maximum sequence length for this model (2100 > 100). Running this sequence through the model will result in indexing errors
..s......Truncation was not explicitly activated but `max_length` is provided a specific value, please use `truncation=True` to explicitly truncate examples to max length. Defaulting to 'longest_first' truncation strategy. If you encode pairs of sequences (GLUE-style) with the tokenizer you can select this strategy more precisely by providing a specific strategy to `truncation`.
/opt/homebrew/lib/python3.11/site-packages/transformers-4.38.0.dev0-py3.11.egg/transformers/tokenization_utils_base.py:2621: FutureWarning: The `pad_to_max_length` argument is deprecated and will be removed in a future version, use `padding=True` or `padding='longest'` to pad to the longest sequence in the batch, or use `padding='max_length'` to pad to a max length. In this case, you can give a specific length with `max_length` (e.g. `max_length=45`) or leave max_length to None to pad to the maximal input size of the model (e.g. 512 for Bert).
  warnings.warn(
.s.s.....FF.Asking to truncate to max_length but no maximum length is provided and the model has no predefined maximum length. Default to no truncation.
...Saving vocabulary to /var/folders/k6/cvg5yzcs1957ywm2m80_y3dw0000gn/T/tmpt4b3xthf/rwkv_vocab_v20230424.txt: vocabulary indices are not consecutive. Please check that the vocabulary is not corrupted!
FSaving vocabulary to /var/folders/k6/cvg5yzcs1957ywm2m80_y3dw0000gn/T/tmpi80ljan7/rwkv_vocab_v20230424.txt: vocabulary indices are not consecutive. Please check that the vocabulary is not corrupted!
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
F..........Saving vocabulary to /var/folders/k6/cvg5yzcs1957ywm2m80_y3dw0000gn/T/tmpuir4d9zz/rwkv_vocab_v20230424.txt: vocabulary indices are not consecutive. Please check that the vocabulary is not corrupted!
......s........Using bos_token, but it is not set yet.
Using eos_token, but it is not set yet.
Using unk_token, but it is not set yet.
Using sep_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using cls_token, but it is not set yet.
Using mask_token, but it is not set yet.
Using additional_special_tokens, but it is not set yet.
..s...
======================================================================
ERROR: test_conversion_reversible (tests.models.rwkv5.test_tokenization_rwkv5.RWKV5TokenizationTest.test_conversion_reversible) [Rwkv5Tokenizer]
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/bbuf/工作目录/RWKV/transformers/tests/test_tokenization_common.py", line 2074, in test_conversion_reversible
    self.assertEqual(tokenizer.convert_tokens_to_ids(word), ind)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/transformers-4.38.0.dev0-py3.11.egg/transformers/tokenization_utils.py", line 649, in convert_tokens_to_ids
    ids.append(self._convert_token_to_id_with_added_voc(token))
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/transformers-4.38.0.dev0-py3.11.egg/transformers/tokenization_utils.py", line 658, in _convert_token_to_id_with_added_voc
    return self._convert_token_to_id(token)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/transformers-4.38.0.dev0-py3.11.egg/transformers/models/rwkv5/tokenization_rwkv5.py", line 153, in _convert_token_to_id
    return self.encoder.get(token.encode("utf-8"), self.unk_token_id)
                            ^^^^^^^^^^^^
AttributeError: 'int' object has no attribute 'encode'

======================================================================
ERROR: test_get_vocab (tests.models.rwkv5.test_tokenization_rwkv5.RWKV5TokenizationTest.test_get_vocab) [Rwkv5Tokenizer]
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/bbuf/工作目录/RWKV/transformers/tests/test_tokenization_common.py", line 2059, in test_get_vocab
    vocab = [tokenizer.convert_ids_to_tokens(i) for i in range(len(tokenizer))]
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/bbuf/工作目录/RWKV/transformers/tests/test_tokenization_common.py", line 2059, in <listcomp>
    vocab = [tokenizer.convert_ids_to_tokens(i) for i in range(len(tokenizer))]
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/transformers-4.38.0.dev0-py3.11.egg/transformers/tokenization_utils.py", line 973, in convert_ids_to_tokens
    return self._convert_id_to_token(ids)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/transformers-4.38.0.dev0-py3.11.egg/transformers/models/rwkv5/tokenization_rwkv5.py", line 157, in _convert_id_to_token
    return self.decoder.get(index, self.unk_token).decode("utf-8")
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'str' object has no attribute 'decode'

======================================================================
FAIL: test_pretokenized_inputs (tests.models.rwkv5.test_tokenization_rwkv5.RWKV5TokenizationTest.test_pretokenized_inputs) [Rwkv5Tokenizer]
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/bbuf/工作目录/RWKV/transformers/tests/test_tokenization_common.py", line 2280, in test_pretokenized_inputs
    self.assertEqual(output, output_sequence)
AssertionError: Lists differ: [66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 7[29 chars], 85] != [66, 33, 67, 68, 69, 70, 71, 72, 73, 74, 75, 7[33 chars], 85]

First differing element 1:
67
33

Second list contains 1 additional elements.
First extra element 20:
85

- [66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85]
+ [66,
+  33,
+  67,
+  68,
+  69,
+  70,
+  71,
+  72,
+  73,
+  74,
+  75,
+  76,
+  77,
+  78,
+  79,
+  80,
+  81,
+  82,
+  83,
+  84,
+  85]

======================================================================
FAIL: test_pretrained_model_lists (tests.models.rwkv5.test_tokenization_rwkv5.RWKV5TokenizationTest.test_pretrained_model_lists)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/bbuf/工作目录/RWKV/transformers/tests/test_tokenization_common.py", line 1027, in test_pretrained_model_lists
    self.assertGreaterEqual(len(self.tokenizer_class.pretrained_vocab_files_map), 1)
AssertionError: 0 not greater than or equal to 1

======================================================================
FAIL: test_save_and_load_tokenizer (tests.models.rwkv5.test_tokenization_rwkv5.RWKV5TokenizationTest.test_save_and_load_tokenizer) [Rwkv5Tokenizer]
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/bbuf/工作目录/RWKV/transformers/tests/test_tokenization_common.py", line 718, in test_save_and_load_tokenizer
    self.assertListEqual(before_tokens, after_tokens)
AssertionError: Lists differ: [0, 0, 0, 0, 45, 0, 0] != [0, 0, 0, 0, 44, 0, 0]

First differing element 4:
45
44

- [0, 0, 0, 0, 45, 0, 0]
?               ^

+ [0, 0, 0, 0, 44, 0, 0]
?               ^


======================================================================
FAIL: test_save_and_load_tokenizer (tests.models.rwkv5.test_tokenization_rwkv5.RWKV5TokenizationTest.test_save_and_load_tokenizer) [Rwkv5Tokenizer]
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/bbuf/工作目录/RWKV/transformers/tests/test_tokenization_common.py", line 743, in test_save_and_load_tokenizer
    self.assertListEqual(before_tokens, after_tokens)
AssertionError: Lists differ: [0, 0, 0, 0, 45, 0, 0] != [0, 0, 0, 0, 44, 0, 0]

First differing element 4:
45
44

- [0, 0, 0, 0, 45, 0, 0]
?               ^

+ [0, 0, 0, 0, 44, 0, 0]
?               ^
```

Alright I’ll checkout the PR to help out with the test ! 

> Alright I’ll checkout the PR to help out with the test !


Thank you, I indeed encountered difficulties while resolving the remaining tests for the tokenizer. I don't quite understand the specific reasons behind these errors. The tests that are throwing these errors are as follows:

```
======================================================================
FAIL: test_conversion_reversible (tests.models.rwkv5.test_tokenization_rwkv5.RWKV5TokenizationTest.test_conversion_reversible) [Rwkv5Tokenizer]
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/bbuf/工作目录/RWKV/transformers/tests/test_tokenization_common.py", line 2075, in test_conversion_reversible
    self.assertEqual(tokenizer.convert_ids_to_tokens(ind), word)
AssertionError: '
' != b'\x00'

======================================================================
FAIL: test_internal_consistency (tests.models.rwkv5.test_tokenization_rwkv5.RWKV5TokenizationTest.test_internal_consistency) [Rwkv5Tokenizer]
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/bbuf/工作目录/RWKV/transformers/tests/test_tokenization_common.py", line 984, in test_internal_consistency
    self.assertEqual(text_2, output_text)
AssertionError: '


' != 'DEF'
- 



+ DEF


======================================================================
FAIL: test_maximum_encoding_length_pair_input (tests.models.rwkv5.test_tokenization_rwkv5.RWKV5TokenizationTest.test_maximum_encoding_length_pair_input) [Rwkv5Tokenizer]
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/bbuf/工作目录/RWKV/transformers/tests/test_tokenization_common.py", line 1253, in test_maximum_encoding_length_pair_input
    self.assertGreater(len(seq0_tokens), 2 + stride)
AssertionError: 0 not greater than 4

======================================================================
FAIL: test_maximum_encoding_length_single_input (tests.models.rwkv5.test_tokenization_rwkv5.RWKV5TokenizationTest.test_maximum_encoding_length_single_input) [Rwkv5Tokenizer]
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/bbuf/工作目录/RWKV/transformers/tests/test_tokenization_common.py", line 1151, in test_maximum_encoding_length_single_input
    self.assertGreater(
AssertionError: 0 not greater than 4 : Issue with the testing sequence, please update it, it's too short

======================================================================
FAIL: test_pretrained_model_lists (tests.models.rwkv5.test_tokenization_rwkv5.RWKV5TokenizationTest.test_pretrained_model_lists)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/bbuf/工作目录/RWKV/transformers/tests/test_tokenization_common.py", line 1027, in test_pretrained_model_lists
    self.assertGreaterEqual(len(self.tokenizer_class.pretrained_vocab_files_map), 1)
AssertionError: 0 not greater than or equal to 1

======================================================================
FAIL: test_save_and_load_tokenizer (tests.models.rwkv5.test_tokenization_rwkv5.RWKV5TokenizationTest.test_save_and_load_tokenizer) [Rwkv5Tokenizer]
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/bbuf/工作目录/RWKV/transformers/tests/test_tokenization_common.py", line 719, in test_save_and_load_tokenizer
    self.assertDictEqual(before_vocab, after_vocab)
AssertionError: {b'\x[115 chars]'': 11, b'\x0b': 12, b'\x0c': 13, b'\r': 14, b[919 chars]': 0} != {b'\x[115 chars]'': 10, b'\x0b': 11, b'\x0c': 12, b'\r': 13, b[918 chars]': 0}
Diff is 4489 characters long. Set self.maxDiff to None to see it.

======================================================================
FAIL: test_save_and_load_tokenizer (tests.models.rwkv5.test_tokenization_rwkv5.RWKV5TokenizationTest.test_save_and_load_tokenizer) [Rwkv5Tokenizer]
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/bbuf/工作目录/RWKV/transformers/tests/test_tokenization_common.py", line 745, in test_save_and_load_tokenizer
    self.assertDictEqual(before_vocab, after_vocab)
AssertionError: {b'\x[115 chars]'': 11, b'\x0b': 12, b'\x0c': 13, b'\r': 14, b[983 chars] 103} != {b'\x[115 chars]'': 10, b'\x0b': 11, b'\x0c': 12, b'\r': 13, b[982 chars] 103}
Diff is 4562 characters long. Set self.maxDiff to None to see it.
```

reproduce command is `python3 -m unittest tests/models/rwkv5/test_tokenization_rwkv5.py` with the pr latest commit.



Having a look right now 🤗 

Created #29095 with the correct tokenizer. note that more tests should be added. 
Do you want me to fully take over to have all the reviews integrated as well? 

> Created #29095 with the correct tokenizer. note that more tests should be added. Do you want me to fully take over to have all the reviews integrated as well?

Thank you. I would be extremely grateful if that were possible. I initially tried to add this model to HuggingFace out of interest, but it seems I underestimated the difficulty. Moreover, due to my own work, the progress on this PR has been very slow. I am very sorry for that.

Alright I'll see what I can do asap

Please note that the implementations for _convert_token_to_id and _convert_id_to_token are implemented kind of backwards and should be more like:
```py
def _convert_token_to_id(self, token):
        """Converts a token (str) in an id using the vocab."""
        return self.decoder.get(token.encode("utf-8"), self.unk_token_id)

def _convert_id_to_token(self, index):
    """Converts an index (integer) in a token (str) using the vocab."""
    return self.encoder.get(index)
```

I'm back from holidays! 🥳 Gotta catch up everything and update the other PR


Nice! I have RWKV 6 ready as well!

On Sun, Mar 17, 2024, 3:57 PM Arthur ***@***.***> wrote:

> I'm back from holidays! 🥳 Gotta catch up everything and update this PR!
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/transformers/pull/26963#issuecomment-2002647017>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ACDK33UMNVLVHGKGN5E6VGTYYYNWVAVCNFSM6AAAAAA6JF7ZBCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMBSGY2DOMBRG4>
> .
> You are receiving this because you commented.Message ID:
> ***@***.***>
>


> I'm back from holidays! 🥳 Gotta catch up everything and update this PR!

Thank you. If you need my assistance, you can call on me at any time. Additionally, I have used the modified `Rwkv5Tokenizer` at https://huggingface.co/RWKV/rwkv-5-world-1b5 and https://huggingface.co/RWKV/rwkv-5-world-3b, and it works well for both individual inference and  batch inference.

@SmerkyG feel free to open a PR for v6 if you want based on #29095

Current use of use_cache seems like it may not be appropriate for RWKV? The state is a required element and cannot really be regenerated slowly, except by essentially using 'use_cache'.

I converted my latest v5 code to match your latest and did a ton of bugfixes such that it now works again in full CPU mode. Please see https://github.com/SmerkyG/RWKV-World-HF-Tokenizer/blob/main/rwkv_world_v5_model_batch/modeling_rwkv5.py to compare for details.

I will put the v6 code in similar shape to prepare for an eventual pull request. 

Also, please note that the CUDA kernel is not currently generating a proper state, so if you batch the initial context and then run with the non-CUDA kernel afterward (which we should really rename to nocuda from cpu) it will give bad results. But pure no-cuda works fine.

> Current use of use_cache seems like it may not be appropriate for RWKV? The state is a required element and cannot really be regenerated slowly, except by essentially using 'use_cache'.
> 
> I converted my latest v5 code to match your latest and did a ton of bugfixes such that it now works again in full CPU mode. Please see https://github.com/SmerkyG/RWKV-World-HF-Tokenizer/blob/main/rwkv_world_v5_model_batch/modeling_rwkv5.py to compare for details.
> 
> I will put the v6 code in similar shape to prepare for an eventual pull request.
> 
> Also, please note that the CUDA kernel is not currently generating a proper state, so if you batch the initial context and then run with the non-CUDA kernel afterward (which we should really rename to nocuda from cpu) it will give bad results. But pure no-cuda works fine.

是的。

> Current use of use_cache seems like it may not be appropriate for RWKV? The state is a required elementow. and cannot really be regenerated slowly, except by essentially using 'use_cache'.
> 
> I converted my latest v5 code to match your latest and did a ton of bugfixes such that it now works again in full CPU mode. Please see https://github.com/SmerkyG/RWKV-World-HF-Tokenizer/blob/main/rwkv_world_v5_model_batch/modeling_rwkv5.py to compare for details.
> 
> I will put the v6 code in similar shape to prepare for an eventual pull request.
> 
> Also, please note that the CUDA kernel is not currently generating a proper state, so if you batch the initial context and then run with the non-CUDA kernel afterward (which we should really rename to nocuda from cpu) it will give bad results. But pure no-cuda works fine.

Could you please submit a fix PR for the implementation [here](https://github.com/BBuf/RWKV-World-HF-Tokenizer/blob/main/rwkv5_model/modeling_rwkv5.py) ? I will review it. Currently, the use_cache indeed does not take effect. And v6 is [here](https://github.com/BBuf/RWKV-World-HF-Tokenizer/blob/main/rwkv6_model/modeling_rwkv6.py), but the result get nan/inf now.

@BBuf Hmm yours has changed a lot since I forked it but is now quite mismatched from @ArthurZucker 's version in this PR. Shouldn't we be moving towards a single unified version? I unfortunately don't have time to do a lot of work on this - I've only been able to do this much because I need it for RWKV6 paper related evals. But I'm happy to share my working v6 (you can already see it in my repo) so you can update yours, and I will do what I can to get v5 cuda working fully with state generation if possible tonight.

working on a PR for you after all :)

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.