Update README.md
Browse files
README.md
CHANGED
@@ -60,7 +60,7 @@ with MosesTokenizer('en') as pretokenize:
|
|
60 |
pretokenized_text = " ".join(pretokenize(text_string))
|
61 |
```
|
62 |
|
63 |
-
Then, to BPE tokenize your text for this model, you should use the [tokenizer trained on Wikitext-103](https://huggingface.co/Kristijan/wikitext-
|
64 |
|
65 |
```python
|
66 |
from transformers import GPT2TokenizerFast
|
|
|
60 |
pretokenized_text = " ".join(pretokenize(text_string))
|
61 |
```
|
62 |
|
63 |
+
Then, to BPE tokenize your text for this model, you should use the [tokenizer trained on Wikitext-103](https://huggingface.co/Kristijan/wikitext-103_tokenizer_v2):
|
64 |
|
65 |
```python
|
66 |
from transformers import GPT2TokenizerFast
|