Distilroberta Tokenizer and Encoder not aligning
Author: dhairyadalal
Pull request: no
Labels: name: Core: Tokenization

I working on a sequence tagging task where I to extract cause and effect sub spans for a given text. For example, extract <e1> and <e2> for the text below.

`"<e2>The Sunshine State drew in a net influx of about $17.7 billion in adjusted gross income (AGI)  -  most of which (72 percent) came from those aged 55 and older.</e2> <e1>It is consistently one of the most popular destinations for retirees due to affordability and low taxes.</e1> Florida's $17.7 billion in net AGI dwarves the remaining 19 states that saw a positive net influx of income  -  which combined for a total of $19.4 billion."`

For the model, I'm try to generate a BIO style tags  that align with the tokenized input from the Distilroberta tokenizer. How I'm finding the after encoding the tokenized input, there is misalignment with the expected tags.

```
text = """The Sunshine State drew in a net influx of about $17.7 billion in adjusted gross income (AGI)  -  most of which (72 percent) came from those aged 55 and older. It is consistently one of the most popular destinations for retirees due to affordability and low taxes. Florida's $17.7 billion in net AGI dwarves the remaining 19 states that saw a positive net influx of income  -  which combined for a total of $19.4 billion."""

cause = 'It is consistently one of the most popular destinations for retirees due to affordability and low taxes.'
effect = 'The Sunshine State drew in a net influx of about $17.7 billion in adjusted gross income (AGI)  -  most of which (72 percent) came from those aged 55 and older.'

# Test that cause and effect are valid substring in text
print(text.find(cause)) # 160
print(text.find(effect)) # 0

from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("distilroberta-base")

# Convert cause into Tags
cause_toks = tokenizer.tokenize(cause)
cause_tags = ["B-cause"] + ["I-cause"] * (len(cause_toks) -1)

# Convert effect into Tags
effect_toks = tokenizer.tokenize(effect)
effect_tags = ["B-cause"] + ["I-cause"] * (len(cause_toks) -1)

# Convert text tokinzed string
text_toks = tokenizer.tokenize(text)
text_toks_string = " ".join(text_toks)
text_toks_string = text_toks_string.replace(" ".join(cause_toks), " ".join(cause_tags))
text_toks_string = text_toks_string.replace(" ".join(effect_toks)," ".join(effect_tags))
text_toks = [tok if tok in ["B-cause", "I-cause", "B-effect", "I-effect"] else "O" for tok in text_toks_string.split()]

#['B-cause', 'I-cause', 'I-cause', 'I-cause', 'I-cause', 'I-cause', 'I-cause', 'I-cause', 'I-cause', 'I-cause', 'I-cause', 'I-cause', 'I-cause', 'I-cause', 'I-cause', 'I-cause', 'I-cause', 'I-cause', 'O', 'I-cause', 'I-cause', 'I-cause', 'I-cause', 'I-cause', 'I-cause', 'I-cause', 'I-cause', 'I-cause', 'I-cause', 'I-cause', 'I-cause', 'I-cause', 'I-cause', 'I-cause', 'I-cause', 'I-cause', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O']


print("text toks len: ", len(text_toks) + 2) # include start and end tokens
print("encoded text len: ", len(tokenizer.encode(text)))
# text toks len:  77
# encoded text len:  100
```

I also found I don't seem to get consistent behavior encoding a text and subspan seperately either.
```
text = "Cats are super coolio"
subtext = "super coolio"

print(tokenizer.encode(text))
# [0, 20913, 32, 2422, 3035, 1020, 2]

print(tokenizer.encode(subtext, add_special_tokens=False))`
# [16101, 3035, 1020]
```

Is my understanding on encode wrong? I thought encode converts the BPE tokens to numerical values and adds the cls and sep tokens according to the beginning and end. But it seems like something else is going on.

Comments:
Dug a bit deeper. It seems encode doesn't deterministically tokenize on white-spaces. In the cat example: 16101 -> "super" and  2422 -> " super". Is there an option in encode to force white space splitting. I guess the hack is to tokenize each word separately but that rather inefficient 

For the GPT2/Roberta tokenizers, the space before a word is part of the word which explain the discrepancy you see.

You can set `add_prefix_space` at initialization, e.g. `tokenizer = AutoTokenizer.from_pretrained("distilroberta-base", add_prefix_space=True)` to always add a space before the text. Performances will be slightly lower as showed in https://github.com/huggingface/transformers/issues/3788 but you will get a consistent behavior when encoding a text and a subspan separately.

I'm not seeing the behavior you described @thomwolf . Perhaps I'm misunderstanding what add_prefix_space does. 

```
tokenizer = AutoTokenizer.from_pretrained("distilroberta-base", add_prefix_space=True)
text = "Cats are super coolio"
subtext = "super coolio"

print(tokenizer.encode(text, add_special_tokens=True))
print(tokenizer.encode(subtext, add_special_tokens=False))
#[0, 20913, 32, **2422, 3035, 1020**, 2]
#[16101, 3035, 1020]
```

```
tokenizer = AutoTokenizer.from_pretrained("distilroberta-base", add_prefix_space=False)
text = "Cats are super coolio"
subtext = "super coolio"

print(tokenizer.encode(text, add_special_tokens=True))
print(tokenizer.encode(subtext, add_special_tokens=False))
# [0, 20913, 32, **2422, 3035, 1020**, 2]
# [16101, 3035, 1020]
```
However, I add special tokens on the subtext, that does seem to work. It just requires extra line parse out the special characters during insertion but I can work with that. This behavior work regardless of whether add_prefix_space is enabled or not.

```
tokenizer = AutoTokenizer.from_pretrained("distilroberta-base")
text = "Cats are super coolio"
subtext = "super coolio"

print(tokenizer.encode(text, add_special_tokens=True))
print(tokenizer.encode(subtext, add_special_tokens=True))

# [0, 20913, 32, 2422, 3035, 1020, 2]
# [0, 2422, 3035, 1020, 2]
```



I see. This is now fixed on master and will be in the next release.
Here is the current behavior with your examples:
```python
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("distilroberta-base", add_prefix_space=True)
text = "Cats are super coolio"
subtext = "super coolio"

print(tokenizer.encode(text, add_special_tokens=True))
[0, 20913, 32, 2422, 3035, 1020, 2]
print(tokenizer.encode(subtext, add_special_tokens=False))
[2422, 3035, 1020]

tokenizer = AutoTokenizer.from_pretrained("distilroberta-base", add_prefix_space=False)
print(tokenizer.encode(text, add_special_tokens=True))
[0, 347, 2923, 32, 2422, 3035, 1020, 2]
print(tokenizer.encode(subtext, add_special_tokens=False))
[16101, 3035, 1020]
```

You can also confirm the behavior with the tokens (the prefix space of the word is this `Ġ` in GPT2/Roberta tokenizers):
```python
tokenizer = AutoTokenizer.from_pretrained("distilroberta-base", add_prefix_space=True)
tokenizer.tokenize(text)
['ĠCats', 'Ġare', 'Ġsuper', 'Ġcool', 'io']
tokenizer.tokenize(subtext)
['Ġsuper', 'Ġcool', 'io']

tokenizer = AutoTokenizer.from_pretrained("distilroberta-base", add_prefix_space=False)
tokenizer.tokenize(text)
['C', 'ats', 'Ġare', 'Ġsuper', 'Ġcool', 'io']
tokenizer.tokenize(subtext)
['super', 'Ġcool', 'io']
```

@thomwolf Thanks! I'll go ahead close this issue. Appreciate the quick turnaround and fix.