Repository: huggingface/tokenizers (language: rust)

Issue:
Incorrect Documentation for tokenizers.pre_tokenizers.ByteLevel?
The documentation for [tokenizers.pre_tokenizers.ByteLevel](https://huggingface.co/docs/tokenizers/python/latest/api/reference.html#tokenizers.pre_tokenizers.ByteLevel) mentions two parameters `add_prefix_space` and `use_regex` of which only one is documented

![image](https://user-images.githubusercontent.com/5028974/163291757-43d06726-ba96-41da-bd73-741cb7d43669.png)

The rust implementation on the other hand appears to have three constructor parameters, `add_prefix_space`, `add_prefix_space`  and `use_regex` 

https://github.com/huggingface/tokenizers/blob/8a9bb28f4667b63f94ed96a3c3672f5db226de4d/tokenizers/src/pre_tokenizers/byte_level.rs#L67

You can pass the three arguments by name to the python api, although if you try and pass them positionally it complains that the ByteLevel constructor only takes a single positional argument.




Discussion / hints:
Thanks for reporting this ! 

We will definitely update this in the docs (the parameters need to be in sync with the signature to contain both `add_prefix_space` and `use_regex`)

The difference between Rust & Python is because Rust uses a single Structure for `decoders.ByteLevel` and `pre_tokenizers.ByteLevel` (so all parameters are needed for the structure, as it implements both traits).

In Python however, we don't expose `trim_offsets` for the `pre_tokenizers.ByteLevel` since it's not actually used to `pre_tokenize`.
(The structure under the hood will contain it, but it's not necessary to do the job).

Fix patch:
diff --git a/bindings/python/py_src/tokenizers/pre_tokenizers/__init__.pyi b/bindings/python/py_src/tokenizers/pre_tokenizers/__init__.pyi
index b149ee171..020559ad3 100644
--- a/bindings/python/py_src/tokenizers/pre_tokenizers/__init__.pyi
+++ b/bindings/python/py_src/tokenizers/pre_tokenizers/__init__.pyi
@@ -100,6 +100,9 @@ class ByteLevel(PreTokenizer):
         add_prefix_space (:obj:`bool`, `optional`, defaults to :obj:`True`):
             Whether to add a space to the first word if there isn't already one. This
             lets us treat `hello` exactly like `say hello`.
+        use_regex (:obj:`bool`, `optional`, defaults to :obj:`True`):
+            Set this to :obj:`False` to prevent this `pre_tokenizer` from using
+            the GPT2 specific regexp for spliting on whitespace.
     """
 
     def __init__(self, add_prefix_space=True, use_regex=True):
diff --git a/bindings/python/src/pre_tokenizers.rs b/bindings/python/src/pre_tokenizers.rs
index 5edc6f2e6..f858ecf75 100644
--- a/bindings/python/src/pre_tokenizers.rs
+++ b/bindings/python/src/pre_tokenizers.rs
@@ -228,6 +228,9 @@ macro_rules! setter {
 ///     add_prefix_space (:obj:`bool`, `optional`, defaults to :obj:`True`):
 ///         Whether to add a space to the first word if there isn't already one. This
 ///         lets us treat `hello` exactly like `say hello`.
+///     use_regex (:obj:`bool`, `optional`, defaults to :obj:`True`):
+///         Set this to :obj:`False` to prevent this `pre_tokenizer` from using
+///         the GPT2 specific regexp for spliting on whitespace.
 #[pyclass(extends=PyPreTokenizer, module = "tokenizers.pre_tokenizers", name=ByteLevel)]
 #[text_signature = "(self, add_prefix_space=True, use_regex=True)"]
 pub struct PyByteLevel {}


Test patch: