Stride in BERT fast tokenizer doesn't work as I expected
Author: mjeensung
Pull request: no
Labels: name: bug

### System Info

Hi, 

I'm using 'bert-base-cased' and 'fast tokenizer'.

I set the stride value as 128, but I found that the stride from tokenized results wasn't 128. In the following reproduction script, the stride between two windows was only 54. Is this a bug or intentional?

### Who can help?

@LysandreJik @SaulLu 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

```
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained('bert-base-cased', do_lower_case=False, use_fast=True)
stride = 128
tokenized_examples = tokenizer(
   ['Big Little Lies (TV series)'],
   [' Despite originally being billed as a miniseries, HBO renewed the series for a second season. Production on the second season began in March 2018 and is set to premiere in 2019. All seven episodes are being written by Kelley and directed by Andrea Arnold. On August 6, 2014, it was announced Nicole Kidman and Reese Witherspoon had optioned the screen rights to Liane Moriarty\'s novel "Big Little Lies". The actresses were expected to develop the project as a film in which they would both star. Bruna Papandrea and Per Saari were set to executive produce alongside Kidman and Witherspoon. Moriarty was also expected to produce as well. On [MASK] 25, 2014, it was announced that Kidman and Witherspoon had decided to develop the project into a limited television series instead of the originally planned film. Additionally, it was announced that television series would be written by David E. Kelley. On May 8, 2015, it was announced that HBO had given the production a series order and that in addition to writing, Kelley would also executive produce. On October 23, 2015, it was reported that Jean-Marc Vallée was in talks to direct the first episode of the series with the potential to direct more. On December 17, 2015, it was announced that Vallée would direct all seven episodes of the series. On November 28, 2016, it was announced that the series would premiere on February 19, 2017.'],
   truncation="only_second" ,
   max_length=192,
   stride=stride,
   return_overflowing_tokens=True,
   return_offsets_mapping=True,
   padding="max_length",
)
pos_mask_window0 = tokenizer.convert_ids_to_tokens(tokenized_examples['input_ids'][0]).index('[MASK]')
pos_mask_window1 = tokenizer.convert_ids_to_tokens(tokenized_examples['input_ids'][1]).index('[MASK]')
print('expected stride: ', stride)
print('actual stride: ', pos_mask_window0 - pos_mask_window1)

>> expected stride: 128
>> actual stride: 54
```

The library versions I'm using are as follows:
transformers                  4.13.0
tokenizers                    0.10.1

### Expected behavior

From the reproduction script above, I expect the observed stride is the same as the defined stride (i.e., 128)

```
>> expected stride: 128
>> actual stride: 128
```

Comments:
Hi @mjeensung,

From my point of view, the result returned by the tokenizer is the one expected given the arguments you specified to the tokenizer, and the stride is of length 128.

 In your example, the tokenized example is composed of a text pair:
- text: 'Big Little Lies (TV series)'
- text pair: ' Despite originally being billed as a miniseries, HBO renewed the series for a second season. Production on the second season began in March 2018 and is set to premiere in 2019. All seven episodes are being written by Kelley and directed by Andrea Arnold. On August 6, 2014, it was announced Nicole Kidman and Reese Witherspoon had optioned the screen rights to Liane Moriarty\'s novel "Big Little Lies". The actresses were expected to develop the project as a film in which they would both star. Bruna Papandrea and Per Saari were set to executive produce alongside Kidman and Witherspoon. Moriarty was also expected to produce as well. On **[MASK]** 25, 2014, it was announced that Kidman and Witherspoon had decided to develop the project into a limited television series instead of the originally planned film. Additionally, it was announced that television series would be written by David E. Kelley. On May 8, 2015, it was announced that HBO had given the production a series order and that in addition to writing, Kelley would also executive produce. On October 23, 2015, it was reported that Jean-Marc Vallée was in talks to direct the first episode of the series with the potential to direct more. On December 17, 2015, it was announced that Vallée would direct all seven episodes of the series. On November 28, 2016, it was announced that the series would premiere on February 19, 2017.'

You ask for only the second sentence to be truncated: the stride will have to be taken from the text pair only. Schematically, the first 2 inputs returned will be:

![image](https://user-images.githubusercontent.com/55560583/179480218-fc205694-8d42-45c5-a7d2-fadecd6796a5.png)

So, to know the stride (based on the mask tokens of your input), I think the formula is : 
```python
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained('bert-base-cased', do_lower_case=False, use_fast=True)
stride = 128

tokenized_examples = tokenizer(
   ['Big Little Lies (TV series)'],
   [' Despite originally being billed as a miniseries, HBO renewed the series for a second season. Production on the second season began in March 2018 and is set to premiere in 2019. All seven episodes are being written by Kelley and directed by Andrea Arnold. On August 6, 2014, it was announced Nicole Kidman and Reese Witherspoon had optioned the screen rights to Liane Moriarty\'s novel "Big Little Lies". The actresses were expected to develop the project as a film in which they would both star. Bruna Papandrea and Per Saari were set to executive produce alongside Kidman and Witherspoon. Moriarty was also expected to produce as well. On [MASK] 25, 2014, it was announced that Kidman and Witherspoon had decided to develop the project into a limited television series instead of the originally planned film. Additionally, it was announced that television series would be written by David E. Kelley. On May 8, 2015, it was announced that HBO had given the production a series order and that in addition to writing, Kelley would also executive produce. On October 23, 2015, it was reported that Jean-Marc Vallée was in talks to direct the first episode of the series with the potential to direct more. On December 17, 2015, it was announced that Vallée would direct all seven episodes of the series. On November 28, 2016, it was announced that the series would premiere on February 19, 2017.'],
   truncation="only_second" ,
   max_length=192,
   stride=stride,
   return_overflowing_tokens=True,
   return_offsets_mapping=True,
   padding="max_length",
)

tokens_ex_0 = tokenizer.convert_ids_to_tokens(tokenized_examples.input_ids[0])
tokens_ex_1 = tokenizer.convert_ids_to_tokens(tokenized_examples.input_ids[1])

sep_position = tokens_ex_0.index("[SEP]")
len_sent_0 = sep_position + 1
pos_mask_window0 = tokens_ex_0.index('[MASK]')
pos_mask_window1 = tokens_ex_1.index('[MASK]')

actual_stride = len(tokens_ex_0) - 1 - pos_mask_window0 + pos_mask_window1 - len_sent_0
```
and the result is 128.

Don't hesitate to tell me if this doesn't answer your question!

Thanks @SaulLu !

I was confused because the tokenized results was different from the ones processed by [squad.py](https://github.com/huggingface/transformers/blob/main/src/transformers/data/processors/squad.py#L187). But I found that [squad.py](https://github.com/huggingface/transformers/blob/main/src/transformers/data/processors/squad.py#L187) re-defines the stride based on max_seq_length and len(truncated_query). 
<img width="803" alt="Screen Shot 2022-07-18 at 11 15 04 PM" src="https://user-images.githubusercontent.com/44629366/179656783-892c2d88-e99c-42d1-a5c1-e69be90f8c61.png">

Thanks for correcting my misunderstanding. Now it's clear to me.