Update README.md
Browse files
README.md
CHANGED
@@ -27,7 +27,7 @@ print(f"Tokens:\n\t{output.input_ids}")
|
|
27 |
|
28 |
## Notes
|
29 |
|
30 |
-
- the default tokenizer (on branch `main`) has a vocab size of
|
31 |
|
32 |
|
33 |
|
@@ -61,7 +61,7 @@ print(f"Tokens:\n\t{output.input_ids}")
|
|
61 |
|
62 |
This should result in the following (_Feb '24 version_):
|
63 |
|
64 |
-
```
|
65 |
>>> print(f"Test string: {test_string}")
|
66 |
Test string: When dealing with Varroa destructor mites, it's crucial to administer the right acaricides during the late autumn months, but only after ensuring that the worker bee population is free from pesticide contamination.
|
67 |
>>>
|
|
|
27 |
|
28 |
## Notes
|
29 |
|
30 |
+
- the default tokenizer (on branch `main`) has a vocab size of 32100
|
31 |
|
32 |
|
33 |
|
|
|
61 |
|
62 |
This should result in the following (_Feb '24 version_):
|
63 |
|
64 |
+
```
|
65 |
>>> print(f"Test string: {test_string}")
|
66 |
Test string: When dealing with Varroa destructor mites, it's crucial to administer the right acaricides during the late autumn months, but only after ensuring that the worker bee population is free from pesticide contamination.
|
67 |
>>>
|