Kristijan commited on
Commit
5667178
1 Parent(s): 744c3c2

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +1 -1
README.md CHANGED
@@ -60,7 +60,7 @@ with MosesTokenizer('en') as pretokenize:
60
  pretokenized_text = " ".join(pretokenize(text_string))
61
  ```
62
 
63
- Then, to BPE tokenize your text for this model, you should use the [tokenizer trained on Wikitext-103](https://huggingface.co/Kristijan/wikitext-103-tokenizer_v2):
64
 
65
  ```python
66
  from transformers import GPT2TokenizerFast
 
60
  pretokenized_text = " ".join(pretokenize(text_string))
61
  ```
62
 
63
+ Then, to BPE tokenize your text for this model, you should use the [tokenizer trained on Wikitext-103](https://huggingface.co/Kristijan/wikitext-103_tokenizer_v2):
64
 
65
  ```python
66
  from transformers import GPT2TokenizerFast