Fast tokenizers calculate wrong offsets when special characters are present
Author: dirkgr
Pull request: no
Labels: name: Core: Tokenization

Example:

```
>>> import transformers
>>> t_fast = transformers.AutoTokenizer.from_pretrained("bert-base-cased", use_fast=True, add_special_tokens=False)
>>> sentence = "A, naïve [MASK] AllenNLP sentence."
>>> tokenized = t_fast.encode_plus(sentence, add_special_tokens=False, return_offsets_mapping=True)
>>> for start, end in tokenized['offset_mapping']:
...     print(repr(sentence[start:end]))
'A'
','
'naïve'
' [MASK'
' Alle'
'nN'
'L'
' sentenc'
'e'
```

As you can see, after the word "naïve", the offsets go off the rails.

Comments:
@mfuntowicz, would it make sense for me to integrate our tokenizer tests into your code, so you can see these things immediately? I'd be happy to do so.

Looks like this is a duplicate of #2917.

On second reading, this is not the same issue as #2917, though they may be related.

Roberta has another related issue:
```
>>> import transformers
>>> t_fast = transformers.AutoTokenizer.from_pretrained("roberta-base", use_fast=True, add_special_tokens=False)
>>> sentence = "I went to the zoo yesterday, but they had only one animal."
>>> tokenized = t_fast.encode_plus(sentence, add_special_tokens=False, return_offsets_mapping=True)
>>> for start, end in (t for t in tokenized['offset_mapping'] if t is not None):
...     print(repr(sentence[start:end]))
'I'
' went'
' to'
' the'
' zoo'
' yesterday'
','
' but'
' they'
' had'
' only'
' one'
' animal'
'.'
```

There are two problems here. `add_special_tokens` is being ignored (#2919), but also, it adds those extra spaces at the front of the words.

Hi @dirkgr, 

Thanks for your report.

Regarding the integration of your tests, it definitively a good idea, if you can put @LysandreJik and myself as reviewers of the PR, we'll have a look 👍.

Regarding `add_special_tokens`, the behaviour on Rust side is quite different as it's a parameter that needs to be provided a construction time, whereas Python allows at tokenisation time. We should make it clearer in the doc.

```python
>>> t = transformers.BertTokenizerFast.from_pretrained('bert-base-cased')
>>> tokenized = t.encode_plus(sentence, add_special_tokens=False, return_offsets_mapping=True)
>>> tokenized['input_ids']
[101, 138, 117, 22607, 103, 4522, 20734, 2101, 5650, 119, 102]

>>> t = transformers.BertTokenizerFast.from_pretrained('bert-base-cased', add_special_tokens=False)
>>> tokenized = t.encode_plus(sentence, add_special_tokens=False, return_offsets_mapping=True)
>>> tokenized['input_ids']
>>> [138, 117, 22607, 103, 4522, 20734, 2101, 5650, 119]
```

For Roberta we're aware of this extra space being included, cc'ing @n1t0 for keeping track of this.

Ok some more context, `GPT2TokenizerFast` has also the same behaviour, so I would extrapolate this occurs more generally on BPE model. 

Adding `<ModelFast>.from_pretrained(..., add_prefix_space=True)` doesn't append the space before but after the token:

```python
'I '
'went '
'to '
'the '
'zoo '
'yesterday,'
' '
'but '
'they '
'had '
'only '
'one '
'animal.'
```

Hi @dirkgr!

So there are multiple different things here:
 - Your first example is due to the space between `naïve` and `[MASK]` not being trimmed out. We are aware of this behavior and are currently working on a fix.
 - The fact that `add_prefix_space=True` moves the space at the end is actually a bug too. This happens because we mess with the offsets while adding the prefix. I am working on a fix for this too.
 - Now, the second example you gave is actually expected behavior:
```python
import transformers
t_fast = transformers.AutoTokenizer.from_pretrained("roberta-base", use_fast=True, add_special_tokens=False)
sentence = "I went to the zoo yesterday, but they had only one animal."
tokenized = t_fast.encode_plus(sentence, return_offsets_mapping=True)
offsets = tokenized['offset_mapping']
tokens = t_fast.tokenize(sentence)
for token, (start, end) in (t for t in zip(tokens, offsets) if t[1] is not None):
    print(repr(token))
    print(repr(sentence[start:end]))
    print(repr(t_fast.decode(t_fast.convert_tokens_to_ids([token]))))
```
will give the following output:
```
'I'
'I'
'I'
'Ġwent'
' went'
' went'
'Ġto'
' to'
' to'
'Ġthe'
' the'
' the'
'Ġzoo'
' zoo'
' zoo'
'Ġyesterday'
' yesterday'
' yesterday'
','
','
','
'Ġbut'
' but'
' but'
'Ġthey'
' they'
' they'
'Ġhad'
' had'
' had'
'Ġonly'
' only'
' only'
'Ġone'
' one'
' one'
'Ġanimal'
' animal'
' animal'
'.'
'.'
'.'
```
Here you can see that the space is actually part of these tokens. That's just the way the byte-level BPE used by GPT-2 and Roberta works. The `Ġ` is actually an encoded space. Does it make sense?

We hit a similar issue when we add new tokens. The input ids are correct, but offsets after the new token are off.

Example: https://colab.research.google.com/drive/1e2a3iyLF9NSMWZR50pnDYRhpizcsRV_6

```
text = "A [test] C"

print(tokenizer.encode(text, add_special_tokens=True))

results = tokenizer.encode_plus(text, 
            return_offsets_mapping=True,
            pad_to_max_length=False,
            max_length=128,
            return_overflowing_tokens=False,
            add_special_tokens=True)

for se in results['offset_mapping']:
  if se:
    print(text[se[0]:se[1]], se)
```

```
[101, 1037, 30522, 1039, 102]
A (0, 1)
 [test (1, 7)
  (8, 9)
```

Potentially related issue huggingface/tokenizers#143


I think all of the mentioned bugs on this issue should now be fixed on `master`

```
>>> import transformers
>>> t_fast = transformers.AutoTokenizer.from_pretrained("bert-base-cased", use_fast=True, add_special_tokens=False)
>>> sentence = "A, naïve [MASK] AllenNLP sentence."
>>> tokenized = t_fast.encode_plus(sentence, add_special_tokens=False, return_offsets_mapping=True)
>>> for start, end in tokenized['offset_mapping']:
...     print(repr(sentence[start:end]))
'A'
','
'naïve'
'[MASK]'
'Allen'
'NL'
'P'
'sentence'
'.'
```
and
```
>>> import transformers
>>> t_fast = transformers.AutoTokenizer.from_pretrained("roberta-base", use_fast=True, add_special_tokens=False)
>>> sentence = "I went to the zoo yesterday, but they had only one animal."
>>> tokenized = t_fast.encode_plus(sentence, add_special_tokens=False, return_offsets_mapping=True)
>>> for start, end in (t for t in tokenized['offset_mapping'] if t is not None):
...     print(repr(sentence[start:end]))
'I'
'went'
'to'
'the'
'zoo'
'yesterday'
','
'but'
'they'
'had'
'only'
'one'
'animal'
'.' 
```
and the last one:
```
from transformers import BertTokenizerFast
tokenizer = BertTokenizerFast.from_pretrained('bert-base-uncased')
tokenizer.add_tokens(['[test]'])
text = "A [test] C"

print(tokenizer.encode(text, add_special_tokens=True))

results = tokenizer.encode_plus(text, 
            return_offsets_mapping=True,
            pad_to_max_length=False,
            max_length=128,
            return_overflowing_tokens=False,
            add_special_tokens=True)

for se in results['offset_mapping']:
  if se:
    print(text[se[0]:se[1]], se)
```
gives
```
[101, 1037, 30522, 1039, 102]
 (0, 0)
A (0, 1)
[test] (2, 8)
C (9, 10)
 (0, 0)
```