`assertion failed: stride < max_len` when using tokenizer with text_pair
Author: zbash-tian
Pull request: no
Labels: name: bug

### System Info

transformers 4.20.1, 4.21.0 (tested with both), MacOS 12.4, Python 3.8.12

### Who can help?

@SaulLu

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

The above occurs when using truncation and overflowing tokens with a sentence pair. Maybe I'm doing something stupid but here is the code that reproduces it:
```
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained('sentence-transformers/msmarco-distilbert-dot-v5')
sentences = [
    "This sentence is not too long but we are going to split it anyway.",
    "This sentence is shorter but will still get split.",
]
inputs = tokenizer(
    sentences[0], sentences[1], truncation=True, return_overflowing_tokens=True, max_length=6, stride=2
)
```

When using the same sentences like this
```
inputs = tokenizer(
    sentences, truncation=True, return_overflowing_tokens=True, max_length=6, stride=2
)
```
all is fine.

### Expected behavior

I would expect that the truncation succeeds as max_length > stride.

Comments:
I'd like to provide an update on my own ticket. Although I still have no idea what causes this behavior and despite probably not being relevant for most practical applications, it seems like the truncation is enforcing that the shortest of the two texts after the truncation is at least 3 tokens long. The resulting error message (see title of this ticket) is only adding to this confusion.
In the following, I'll describe the experiments I ran and the outcomes that led me to the aforementioned conclusion.

First, I determined the lengths of the above sentences after tokenization.
```
len(tokenizer(sentences[0])['input_ids'])  # 17 when `add_special_tokens=True`, 15 otherwise
len(tokenizer(sentences[1])['input_ids'])  # 12 when `add_special_tokens=True`, 10 otherwise
```

Then I played around with the `max_length` parameter to the tokenizer's `__call__` method to determine if there's a setting which lets it complete the tokenization.
For
```
inputs = tokenizer(
    sentences[0], sentences[1], truncation="longest_first", return_overflowing_tokens=True, max_length=6, stride=2,
    add_special_tokens=True
)
```
it was `max_length = 6` and for
```
inputs = tokenizer(
    sentences[0], sentences[1], truncation="longest_first", return_overflowing_tokens=True, max_length=9, stride=2,
    add_special_tokens=True
)
```
it was `max_length = 9`, showing the impact of the 3 special tokens added before, between, and after the two sequences.

Then I tried the same experiment for `truncation="only_second"` which has different `max_length` thresholds, as the first sequence needs to fit completely. This resulted in settings of `max_length = 18` and `max_length = 21` for `add_special_tokens=False` and `add_special_tokens=True`, respectively.

Based on these observations, I concluded, as mentioned in the introduction, that the shortest sequence must have at least 3 tokens after the truncation for it to be successful. I'm wondering about 2 things now:
1. Should the documentation be adjusted to make this specific?
2. Should the error message be improved as it was misleading, at least to me.

Thanks for your support!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.