Repository: huggingface/tokenizers (language: rust)

Issue:
Error message for too-high stride could be much clearer
### System Info

- `transformers` version: 4.30.0.dev0
- Platform: macOS-10.16-x86_64-i386-64bit
- Python version: 3.10.11
- Huggingface_hub version: 0.13.4
- Safetensors version: not installed
- PyTorch version (GPU?): 2.0.0 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no

### Who can help?

@ArthurZucker

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

## Input:

I ran my Python interpreter with an env variable set to show a full Rust backtrace:

```bash
RUST_BACKTRACE=full python
```

then tried to tokenize a long text with a value of `stride` that I think would seem reasonable to many users:

```python
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained('bert-base-cased')
tokenizer("Hello, world. " * 1000, stride=510, truncation=True)
```

## Output

I received the following exeception:

```
thread '<unnamed>' panicked at 'assertion failed: stride < max_len', /Users/runner/work/tokenizers/tokenizers/tokenizers/src/tokenizer/encoding.rs:319:9
stack backtrace:
   0:        0x12618ad36 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::h47bed05ceb7970ee
   1:        0x1261a67ca - core::fmt::write::hefd823d99333384a
   2:        0x12618594c - std::io::Write::write_fmt::hb978956bb1b85e2b
   3:        0x12618ab1a - std::sys_common::backtrace::print::hc9e298c4664da113
   4:        0x12618c753 - std::panicking::default_hook::{{closure}}::hb97411d6e916e1d2
   5:        0x12618c4a8 - std::panicking::default_hook::hc3ac81176817192f
   6:        0x12618ce8f - std::panicking::rust_panic_with_hook::hc11209fa2686218a
   7:        0x12618cc13 - std::panicking::begin_panic_handler::{{closure}}::h5bad1bd665ab68a6
   8:        0x12618b1d8 - std::sys_common::backtrace::__rust_end_short_backtrace::h6f4efa73918edf57
   9:        0x12618c91d - _rust_begin_unwind
  10:        0x1261da433 - core::panicking::panic_fmt::hb5474ac70e328787
  11:        0x1261da4c7 - core::panicking::panic::hff10b945c4ba062c
  12:        0x125e09f79 - tokenizers::tokenizer::encoding::Encoding::truncate::hb38557859664465b
  13:        0x125e35db9 - tokenizers::utils::truncation::truncate_encodings::h8205a880294e252e
  14:        0x125d6be8b - tokenizers::tokenizer::TokenizerImpl<M,N,PT,PP,D>::post_process::h9748c77ab5f09cae
  15:        0x125d6d462 - tokenizers::tokenizer::TokenizerImpl<M,N,PT,PP,D>::encode_char_offsets::h631a29c4abd0f034
  16:        0x125d20bb0 - <alloc::vec::Vec<T,A> as alloc::vec::spec_extend::SpecExtend<T,I>>::spec_extend::h8db37c338ca0385c
  17:        0x125d81019 - rayon::iter::plumbing::bridge_producer_consumer::helper::h2a6fe3a9ec158247
  18:        0x125ce3e8e - <rayon::vec::IntoIter<T> as rayon::iter::IndexedParallelIterator>::with_producer::hefedf78310c9d1ae
  19:        0x125d1d21d - rayon::iter::extend::<impl rayon::iter::ParallelExtend<T> for alloc::vec::Vec<T>>::par_extend::hb2928a26bd73a234
  20:        0x125cb4b80 - rayon::result::<impl rayon::iter::FromParallelIterator<core::result::Result<T,E>> for core::result::Result<C,E>>::from_par_iter::hcf438eebac8e47cf
  21:        0x125d6daed - tokenizers::tokenizer::TokenizerImpl<M,N,PT,PP,D>::encode_batch_char_offsets::h9b7bd541371d7730
  22:        0x125d0ecbd - pyo3::marker::Python::allow_threads::h96e54142caac5333
  23:        0x125c53e37 - tokenizers::tokenizer::PyTokenizer::encode_batch::hcd6e834d07735917
  24:        0x125c59f72 - tokenizers::tokenizer::_::<impl tokenizers::tokenizer::PyTokenizer>::__pymethod_encode_batch__::hf054979f2c2e161b
  25:        0x125c466d2 - pyo3::impl_::trampoline::trampoline_inner::ha84889760a7ebfe4
  26:        0x125c546e4 - tokenizers::tokenizer::_::<impl pyo3::impl_::pyclass::PyMethods<tokenizers::tokenizer::PyTokenizer> for pyo3::impl_::pyclass::PyClassImplCollector<tokenizers::tokenizer::PyTokenizer>>::py_methods::ITEMS::trampoline::h98e5dfb88740eead
  27:        0x10ec62801 - _cfunction_vectorcall_FASTCALL_KEYWORDS
  28:        0x10ed4a61b - _call_function
  29:        0x10ed1b4ae - __PyEval_EvalFrameDefault
  30:        0x10ec04326 - __PyFunction_Vectorcall
  31:        0x10ec08dee - _method_vectorcall
  32:        0x10ed2a2b4 - __PyEval_EvalFrameDefault
  33:        0x10ec04326 - __PyFunction_Vectorcall
  34:        0x10ec08dee - _method_vectorcall
  35:        0x10ed2a2b4 - __PyEval_EvalFrameDefault
  36:        0x10ec04326 - __PyFunction_Vectorcall
  37:        0x10ec08dee - _method_vectorcall
  38:        0x10ed2a2b4 - __PyEval_EvalFrameDefault
  39:        0x10ec04326 - __PyFunction_Vectorcall
  40:        0x10ec08dee - _method_vectorcall
  41:        0x10ed2a2b4 - __PyEval_EvalFrameDefault
  42:        0x10ec04326 - __PyFunction_Vectorcall
  43:        0x10ec05faf - __PyObject_Call_Prepend
  44:        0x10ec908ae - _slot_tp_call
  45:        0x10ed4a7dd - _call_function
  46:        0x10ed1b4ae - __PyEval_EvalFrameDefault
  47:        0x10ed17901 - __PyEval_Vector
  48:        0x10eda0360 - _run_mod
  49:        0x10eda06ef - _PyRun_InteractiveOneObjectEx
  50:        0x10ed9f9aa - __PyRun_InteractiveLoopObject
  51:        0x10ed9f54f - __PyRun_AnyFileObject
  52:        0x10eda191a - _PyRun_AnyFileExFlags
  53:        0x10edc437f - _pymain_run_stdin
  54:        0x10edc3b57 - _pymain_run_python
  55:        0x10edc38d5 - _Py_RunMain
  56:        0x10eb9f7c8 - _main
  57:     0x7ff80b8b841f - <unknown>
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<HOME_DIR>/PycharmProjects/transformers/src/transformers/tokenization_utils_base.py", line 2548, in __call__
    encodings = self._call_one(text=text, text_pair=text_pair, **all_kwargs)
  File "<HOME_DIR>/PycharmProjects/transformers/src/transformers/tokenization_utils_base.py", line 2654, in _call_one
    return self.encode_plus(
  File "<HOME_DIR>/PycharmProjects/transformers/src/transformers/tokenization_utils_base.py", line 2727, in encode_plus
    return self._encode_plus(
  File "<HOME_DIR>/PycharmProjects/transformers/src/transformers/tokenization_utils_fast.py", line 497, in _encode_plus
    batched_output = self._batch_encode_plus(
  File "<HOME_DIR>/PycharmProjects/transformers/src/transformers/tokenization_utils_fast.py", line 425, in _batch_encode_plus
    encodings = self._tokenizer.encode_batch(
pyo3_runtime.PanicException: assertion failed: stride < max_len
```

### Expected behavior

In some cases, the `stride < max_len` condition can fail because there are too many special tokens added by the tokenizer, which count against the "effective max length" of the model/tokenizer (e.g. BERT has a model max length of 512, but its tokenizer inserts two special characters by default, so the stride can be at most `512 total tokens - 2 special tokens - (at least) 1 token of movement per window = 509 tokens`).

Users are much more likely to run into this difficult-to-understand error message because Huggingface's definition of "stride" is the opposite of how it is defined in other packages (e.g. [convolutional layers in TensorFlow](https://www.kaggle.com/code/ryanholbrook/the-sliding-window#Stride), where it defines how much the sliding window moves at each step, not how much overlap there is). So a user who wants "no overlap" between windows would be very likely to try to set the `stride` to the max length of the model (`512` for BERT), and might be confused by an error message saying that `stride < max_len` is a requirement. After reviewing the documentation and understanding how `stride` works, the user might realize that they actually want `stride=0`, but it would be very confusing why a `stride=511` or even `stride=510` won't work.


Discussion / hints:
cc @ArthurZucker 
Hey! Thanks for posting. 
If you are suggesting to update the documentation, feel free to open a PR! As long as the definition that we are using is properly documented, I don't think that we should change it just to match other libraries. 
@ArthurZucker Transformers' definition of "stride" being different from other tools' was not the main point of this issue. The problem is that `stride < max_len` is a very opaque error message. The user has no way of knowing that `max_len` is not the _original_ maximum length of the model (e.g. 512 tokens for BERT), but rather the "_effective_" maximum length after subtracting special tokens (e.g. 510 for BERT).

It appears that this error message is ultimately coming from some Rust code; I'm not familiar with Rust but it looks like the "panic" is ultimately propagating from [this line](https://github.com/huggingface/tokenizers/blob/c7102c4c0fc3b1c6efd3d1ee1b0a80fe966cd504/tokenizers/src/tokenizer/encoding.rs#LL319C9-L319C16).

- Is there any way to get a better error message (i.e. include a parenthetical saying something like `(max_len is the effective max length of the model, after subtracting special tokens)`)
- Should this actually be an issue in the [tokenizers repo](https://github.com/huggingface/tokenizers)? I'm now starting to think that would be a more appropriate place, but it doesn't have a contribution guide, so I'm not sure
Yep! If this is originating from rust code, let's move it to rust! 
I 'll open a PR to fix the error message! 

Fix patch:
diff --git a/tokenizers/src/tokenizer/encoding.rs b/tokenizers/src/tokenizer/encoding.rs
index 40576efea..0693ad1e1 100644
--- a/tokenizers/src/tokenizer/encoding.rs
+++ b/tokenizers/src/tokenizer/encoding.rs
@@ -316,7 +316,7 @@ impl Encoding {
             return;
         }
 
-        assert!(stride < max_len);
+        assert!(stride < max_len, "`stride` must be strictly less than `max_len={}` (note that `max_len` may be shorter than the max length of the original model, as it subtracts the number of special characters", max_len);
 
         // When truncating, we lose the `sequence_ranges` information.
         self.sequence_ranges.clear();


Test patch: