How can i use bpe tokenizer  in t5 pretrain from scratch 
Author: 520jefferson
Pull request: no
Labels: name: bug

### System Info

```shell
transformers version: 4.20.0.dev0
when i use the transformers/examples/flax/language-modeling/run_t5_mlm_flax.py to pretrain the t5 from scratch, the preprocess i use the bpe codes to split sentence but not the original tokenizer. How can i use the bpe codes?
```


### Who can help?
@patrickvonplaten
@SaulLu


### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

![image](https://user-images.githubusercontent.com/5691554/171125809-c210799b-46f5-4657-b598-709e663f4885.png)


### Expected behavior

```shell
replace the tokenizer with load bpe codes to split sentence
```


Comments:
Hey @520jefferson,

Could you please provide us with a codesnippet that we can copy-paste directly into a terminal? We sadly don't have the time to try to guess what we should reproduce.

Thanks a lot!

Hey @patrickvonplaten 
i just want to  make sure whether the run_t5_mlm_flax.py script provider another tokenizer  which can just load bpe codes or vocab to initialize the tokenizer. 

Or  maybe i don't want to use tokenizer, i just need vocab.txt,  because i can preprocess with bpe tokenizer before training. How should i do my training without tokenizer?

Thanks


Think you have to use a tokenizer to do training (the model needs numbers not letters). You could otherwise try to use `Canine` or `ByT5`

@patrickvonplaten 

Tokenizer can just split the sentence according to space like txts = text.split(" ") , and the token txts[i] can find in vocab.txt, then it can be transfer to number, so i just need the tokenizer load vocab.txt and split according to space , then they can be transfer to numbers.

Thanks for reply !


Hi!

If you're ever sure you want to do this (you probably know this, but there are an infinite number of possible words and the size of the vocabulary is computationally expensive), you can create this type of tokenizer with the tokenizers library by instantiating a particular [WordLevel](https://huggingface.co/docs/tokenizers/api/models#tokenizers.models.WordLevel) model. Then you will have to load it in a `PreTrainedTokenizerFast` of transformers. All the necessary steps are explained [here](https://huggingface.co/docs/transformers/fast_tokenizers).

Hi, @SaulLu 

1. I set like like:
**>>> from transformers import PreTrainedTokenizerFast
>>> from tokenizers.models import WordLevel
>>> vocab = WordLevel.from_file("./chitchat-t5-base/vocab.json","<unk>")
>>> fast_tokenizer = PreTrainedTokenizerFast(tokenizer_object=vocab)
>>> fast_tokenizer.decode("平时 你 干点 什么 <sep> 找 工作",skip_special_tokens=True)
**
Then i got this: AttributeError: 'tokenizers.models.WordLevel' object has no attribute 'decode':
![image](https://user-images.githubusercontent.com/5691554/172313182-7c55f31b-fdea-43c8-9bfc-6a3babd5904c.png)
and i also use tokenize, it sames i don't set the trunctation:
![image](https://user-images.githubusercontent.com/5691554/172316728-f04da1e1-e293-475d-b769-fd77d7b71341.png)
and i use encode, i met this(**_AttributeError: 'tokenizers.models.WordLevel' object has no attribute 'truncation_**'):
![image](https://user-images.githubusercontent.com/5691554/172339936-b5f6e28f-068c-45b5-9ba1-4f21f0731e17.png)


2. Or if i replace autotokenizer to PreTrainedTokenizerFast in run_t5_mlm_flax.py, and train a new t5, i got this:
![image](https://user-images.githubusercontent.com/5691554/172313489-72bd400a-2850-40e8-ab5c-e94290f8ab90.png)

3.My vocab.json like this below, it seems the tokenizer cannot load this vocab. In order to use the tokenizer how should i load this to a proper tokenizer? 
{
"<unk>": 0,
"<eod>": 1,
"<pad>": 2,
"<mask>": 3,
"<sep>": 4,
"，": 5,
"的": 6,
"？": 7,
"了": 8,
.....
.....
.....
"<s_181>": 33786,
"<s_182>": 33787,
"<s_183>": 33788,
"<s_184>": 33789,
"<s_185>": 33790,
"<s_186>": 33791,
"<s_187>": 33792,
"<s_188>": 33793,
"<s_189>": 33794
}

4. i also try like this,  it doesno't work:
![image](https://user-images.githubusercontent.com/5691554/172549622-f1a84cca-b5be-4f92-9dd9-14179520d0a6.png)
then i uninstall transformers and sentencepiece and pip install transformers sentencepiece, the errors are same. 
![image](https://user-images.githubusercontent.com/5691554/172586169-ac8185d3-adc9-4c23-be94-dc5e01ec3d5b.png)



Hey @520jefferson,

Could you please try to to not post screenshots here? Not that we cannot reproduce code from screenshots as it's not easily copy-pastable into the command line

@patrickvonplaten @SaulLu 

Thanks for reply,  codes and vocab like follows.

1. The vocab.txt (vocab.json is manually constructed from vocab.txt ) and meregs.txt i upload to google drive as follows:
vocab.txt:https://drive.google.com/file/d/10jC8L_-RDLRv5QkAato8nJWGU1UQQcz1/view?usp=sharing
vocab.json:https://drive.google.com/file/d/1e5Ll0bAHhikhnYV5XaW3NB8aTSWdCvnC/view?usp=sharing
merges.txt:https://drive.google.com/file/d/1ifXlQaYod_kobqgNe82tmHTtHpxBYnBq/view?usp=sharing

2.The sentences for  training and validation and test like this (after bpe, tokens split by " "):
你 觉得 大人 辛苦 还是 学生 辛苦 <sep> 都 很 辛苦
头条 文章 没 啥 违规 ， 却 被 小@@ 浪@@ 浪 屏蔽 了 ， 而且 删 了 先生 的 转发 评价 ， 农历 新年 将 至 ， 俺 不想 发火 ， 行 ， 俺 再 发 一遍 ！ <sep> 怎么 删 了 ， 还 没 看 呢
专辑 有 签名 么 ？ ！ … <sep> 没有 机会 去 签@@ 售@@ 会 啦 幸好 里面 的 容 和 小 卡片 有 签名
你 帮 我 买 东西 吗 <sep> 你 给钱 我 ， 当然 帮 你 买 耶
你 说 那个 早晨 喝 那个 水有 什么 好处 <sep> 可以 提高 睡眠 质量 <sep> 养成 良好 的 睡眠 时间 和 习惯 <sep> 慢慢 养成 早睡早起 的 习惯 ， 习@@ 惯@@ 成@@ 自然
求个 风景 超 美的 网游 最好 是 韩国 的 <sep> 剑侠情缘 叁
现在 百度 帐号 是 不能 拿 邮箱 注册 了 么 ？ 只能 拿 手机号 了 么 ？ 如果 可以 应该 怎么 拿 邮箱 注册 ？ 谢谢 ！ <sep> 先 用 手机 注册 ， 然后 绑定 一个 邮箱 ， 再@@ 解 绑 手机 即可
咱们 出去 转 会儿 遛@@ 弯@@ 儿 去 呗 <sep> 我 在 工@@ 体 的 漫 咖啡 ， 要 不要 来 坐 会儿
我 知道 最近 做 什么 <sep> 准备 演唱会 的 事 吧

3.i want to use the tokenizer to load the vocab and tokenizer to tokenizer my sentence and give it to the t5 model.
load model like this(config: https://drive.google.com/file/d/1WOb-gqjkt1m6GBTFeq4wOWS3dW3Qt1oK/view?usp=sharing):
from transformers import T5Config, T5ForConditionalGeneration
config = T5Config.from_json_file(config_file)
model = T5ForConditionalGeneration(config)

load tokenizer:
from tokenizers.models import WordLevel
from transformers import PreTrainedTokenizerFast
vocab = WordLevel.from_file("vocab.json","<unk>")
fast_tokenizer=PreTrainedTokenizerFast(tokenizer_object=vocab)
fast_tokenizer.encode("你 觉得 大人 辛苦 还是 学生 辛苦 <sep> 都 很 辛苦")

then i met this errror:AttributeError: 'tokenizers.models.WordLevel' object has no attribute 'truncation'

4. So i want to load the vocab into tokenizer and use it like this { source = tokenizer.batch_encode_plus([source_text], max_length= 75, pad_to_max_length=True, truncation=True, padding="max_length", return_tensors='pt')
 }   and return { 'source_ids': source_ids.to(dtype=torch.long),     'source_mask': source_mask.to(dtype=torch.long),          'target_ids': target_ids.to(dtype=torch.long),  'target_ids_y': target_ids.to(dtype=torch.long) }  ,  and give the tokenizer result to model and train the model like translation task, how should i do ?






This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.