Add readme
Browse files
README.md
ADDED
|
@@ -0,0 +1,7 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
This repository doesn't contain a model, but only a tokenizer that can be used with the
|
| 2 |
+
`tokenizers` library.
|
| 3 |
+
|
| 4 |
+
```python
|
| 5 |
+
from tokenizers import Tokenizer
|
| 6 |
+
tokenizer = Tokenizer.from_pretrained("anthony/tokenizers-test")
|
| 7 |
+
```
|