Subword Tokenization Bug after Non-Space Word Boundaries (AlbertTokenizerFast)
Author: mcrchopra
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.11.0
- Platform: macOS Big Sur
- Python version: 3.8.5

### Who can help
@LysandreJik

## Information

When using the AlbertTokenizer or AlbertTokenizerFast, words that appear right after word boundaries (except spaces) are not tokenized properly. I have been able to replicate this problem for word boundaries such as brackets `[{(`, slashes `/` or dashes `-`, but there may be others.

 However, if a space is introduced between these word boundaries and the subsequent word, the phrase is tokenized properly. For example, `(presentation)` is tokenized incorrectly by AlbertTokenzerFast, while `( presentation)` is tokenized correctly by AlbertTokenzerFast. This issue doesn't exist for BertTokenizerFast (see detailed example below).

It seems like words that follow non-space word boundaries are treated as a continuation of the word boundary. I would expect that the word would be treated as a new word. This leads to incorrect subword tokenization.  For example, AlbertTokenizerFast is tokenizing `(presentation)` with the subword tokens `present, ation` rather than with the start word token `▁presentation`.

Model I am using: **ALBERT**

The problem arises when using:
* [x] my own modified scripts

## To reproduce

Steps to reproduce the behavior:

``` python
from transformers import AlbertTokenizerFast, BertTokenizerFast
test_phrase = "Knowledge of (presentation) would be great."
albert_tokenizer = AlbertTokenizerFast.from_pretrained('albert-base-v2')
bert_tokenizer = BertTokenizerFast.from_pretrained('bert-base-uncased')
albert_tokenizer_tokens = albert_tokenizer(test_phrase).tokens()
bert_tokenizer_tokens = bert_tokenizer(test_phrase).tokens()

print("Albert Tokenizer Tokens: {}".format(albert_tokenizer_tokens))
print("Bert Tokenizer Tokens: {}".format(bert_tokenizer_tokens))
```

The print statements produce the following:
```
Albert Tokenizer Tokens: ['[CLS]', '▁knowledge', '▁of', '▁', '(', 'present', 'ation', ')', '▁would', '▁be', '▁great', '.', '[SEP]']
Bert Tokenizer Tokens: ['[CLS]', 'knowledge', 'of', '(', 'presentation', ')', 'would', 'be', 'great', '.', '[SEP]']
```

Notably, the word that appears after the parenthesis, presentation, is tokenized into the tokens `present` and `ation`, rather than `▁presentation`. 

## Expected behavior

I would expect the print statements to produce the following:
```
Albert Tokenizer Tokens: ['[CLS]', '▁knowledge', '▁of', '▁', '(', '▁presentation', ')', '▁would', '▁be', '▁great', '.', '[SEP]']
Bert Tokenizer Tokens: ['[CLS]', 'knowledge', 'of', '(', 'presentation', ')', 'would', 'be', 'great', '.', '[SEP]']
```

<!-- A clear and concise description of what you would expect to happen. -->


Comments:
cc @SaulLu 

Hi @mcrchopra ,

Could you tell us more about why you think the tokenization of `(presentation)` with Albert's tokenizer should be `'(', '▁presentation', ')'` and not `'(', 'present', 'ation', ')'`? 

Apriori, there is no reason to have similar tokenizations produced with Albert's and Bert's tokenizers. From my point of view, the tokenization returned by the Albert tokenizer corresponds well to the rules we gave it but I would be really keen to know your opinion on it.  :blush:

Hi @SaulLu, thanks for the quick response! 

This specifically becomes an issue, when you introduce new tokens to the Albert vocabulary and continue pre-training. For the sake of continuity, let's say that the word "presentation" (from the example above) was not present in the original vocabulary. 

Given the current behavior, phrases with `(presentation)` would tokenize the term into subwords rather than use the new token `presentation`. This defeats the purpose of adding the new token altogether (i.e. lessens the amount of training data that the model has in it's purview). 

In the dataset I'm using to further pre-train Albert, there are a large number of parentheticals and slashes, which is "reducing the training signal".

Hi @mcrchopra , thank you for detailing your use case!


> Given the current behavior, phrases with (presentation) would tokenize the term into subwords rather than use the new token presentation. This defeats the purpose of adding the new token altogether (i.e. lessens the amount of training data that the model has in it's purview).

Could you share with me what you did to add the token `presentation` to your tokenizer? :relaxed:  On my side, if I realize this addition to the tokenizer of "albert-base-v2", the token `presentation` is well preserved on the example that you shared:

```python
from transformers import AlbertTokenizerFast

tokenizer = AlbertTokenizerFast.from_pretrained("albert-base-v2")

text = "Knowledge of (presentation) would be great."
tokenizer.tokenize(text)
```
output
```
['▁knowledge', '▁of', '▁', '(', 'present', 'ation', ')', '▁would', '▁be', '▁great', '.']
 ```
 
 and here's the result after I add the token:
```python
tokenizer.add_tokens(["presentation"])
tokenizer.tokenize(text)
```
output
```
['▁knowledge', '▁of', '▁', '(', 'presentation', '▁', ')', '▁would', '▁be', '▁great', '.']
 ```

Hi @SaulLu, thank you so much for looking into this, especially before the holidays!

I was wrong -- as you show your in your example, this totally works for added tokens.

To provide a little more context, in my use case I am training ALBERT on a dataset from a different domain. Within the new domain, there were certain tokens that  were already are captured in the vocabulary of the original ALBERT tokenizer.  

I was **not adding these tokens** via the "add_tokens" method.  It seems like these tokens are the ones having the issue I presented above.  My hypothesis is that these tokens are quite in-frequent in the original data distribution upon which ALBERT was trained; so the tokenizer uses subword representations rather than the full word, when word boundaries are present. 

For now, my solution will be to add these overlapping tokens as new tokens, essentially forcing the model to re-learn the embeddings for these tokens, on my given domain.

Again, thanks so much for looking into this! You're feedback help me figure it out :) 

I'm glad to read that it helped you! :hugs: 

If I understood your use case correctly: you plan to modify the tokenizer a bit and to continue training the model a bit on this modified tokenizer. If that is the case, I think you can have I have another solution to show you. Of course, adding tokens with the `add_tokens` method may be the best solution for in the end, but I just want to show you an alternative (which necessarily requires some re-training of your model).

Actually, if your goal is that the text contained between parentheses is treated as text that would be at the beginning of a sentence, then you can also change the pre-tokenization rules of your tokenizer in order to "force" a division at the level of parentheses.


Here is how to do it (if it is more commanding you can also directly open [this google colab](https://colab.research.google.com/drive/16XLwof-eCRMJZRIutT2n9bDltRPTMRAq?usp=sharing)):
### A. We load the original albert tokenizer


```python
import json

from transformers import AlbertTokenizerFast
from tokenizers import pre_tokenizers
```

```python
text = "Knowledge of (presentation) would be great."
```


```python
tokenizer = AlbertTokenizerFast.from_pretrained("albert-base-v2")
```

```python
print(tokenizer.tokenize(text))
```

    ['▁knowledge', '▁of', '▁', '(', 'present', 'ation', ')', '▁would', '▁be', '▁great', '.']

We store the encoding of this sentence in order to compare it at the end with the encoding of the same sentence done with our new tokenizer

```python
ids_tokenization_original_tokenizer = tokenizer.encode(text)
ids_tokenization_original_tokenizer_map = [(id, tokenizer.convert_ids_to_tokens([id])) for id in ids_tokenization_original_tokenizer]
print(ids_tokenization_original_tokenizer_map)
```

    [(2, ['[CLS]']), (1918, ['▁knowledge']), (16, ['▁of']), (13, ['▁']), (5, ['(']), (3914, ['present']), (857, ['ation']), (6, [')']), (83, ['▁would']), (44, ['▁be']), (374, ['▁great']), (9, ['.']), (3, ['[SEP]'])]


### B. We adapt the tokenizer to treat the content of parenthesis as a regular sentence beginning with a space

1. We add a rule to pre-tokenize the text on parenthesis


```python
parenthesis_1 = "("
parenthesis_2 = ")"
behavior = "isolated"

tokenizer.backend_tokenizer.pre_tokenizer = pre_tokenizers.Sequence(
            [
                pre_tokenizers.WhitespaceSplit(),
                pre_tokenizers.Split(parenthesis_1, behavior=behavior),
                pre_tokenizers.Split(parenthesis_2, behavior=behavior),
                pre_tokenizers.Metaspace(replacement="▁", add_prefix_space=True),
            ]
        )
```

2. We test the result


```python
print(tokenizer.tokenize(text))
```

    ['▁knowledge', '▁of', '▁', '(', '▁presentation', '▁', ')', '▁would', '▁be', '▁great', '.']


We observe here that `" ("` is tokenized into 2 tokens `['▁', '(']`. This is probably due to the fact that the tokenizer vocabulary was not learned with this pre-tokenization rule. We can tweak the vocabulary to solve this problem.


```python
tokenizer.save_pretrained('local')
```

3. We change the vocabulary to replace the token `"("` with the token `"▁("` and `")"` with `"▁)"` because with the new pre-tokenization rule, a parenthesis will always be preceded by a space. 


```python
with open('local/tokenizer.json', "r") as f:
    tokenizer_json = json.loads(f.read())

vocab = tokenizer_json['model']['vocab']
```

we first verify that the tokens `"▁("` and `"▁)"` are not contained into our vocabulary to confirm our guess.


```python
tokens_to_check = ["▁(", "▁)"]
for token, prob in vocab:
    if token in  tokens_to_check:
        print(token)
```

Now we are sure that `"▁("` and `"▁)"` are not in the vocabulary, we can do the replacement (in order to continue to use the same id!)


```python
new_vocab = []
tokens_to_change = ["(", ")"]
for token, prob in vocab:
    if token in  tokens_to_change:
        token = f"▁{token}"
    new_vocab.append([token, prob])

tokenizer_json['model']['vocab'] = new_vocab

with open('local/tokenizer.json', "w") as f:
    f.write(json.dumps(tokenizer_json))
```

4. That's it, we can load our new tokenizer and compare the tokenization with the original one!


```python
tokenizer = AlbertTokenizerFast.from_pretrained('local')

print(tokenizer.tokenize(text))
```

    ['▁knowledge', '▁of', '▁(', '▁presentation', '▁)', '▁would', '▁be', '▁great', '.']


We do a last check to verify that we didn't mess with the ids (which is the thing that matter to the model)


```python
ids_tokenization_new_tokenizer = tokenizer.encode(text)
ids_tokenization_new_tokenizer_map = [(id, tokenizer.convert_ids_to_tokens([id])) for id in ids_tokenization_new_tokenizer]

print(ids_tokenization_original_tokenizer_map)
print(ids_tokenization_new_tokenizer_map)
```

    [(2, ['[CLS]']), (1918, ['▁knowledge']), (16, ['▁of']), (13, ['▁']), (5, ['(']), (3914, ['present']), (857, ['ation']), (6, [')']), (83, ['▁would']), (44, ['▁be']), (374, ['▁great']), (9, ['.']), (3, ['[SEP]'])]
    [(2, ['[CLS]']), (1918, ['▁knowledge']), (16, ['▁of']), (5, ['▁(']), (6364, ['▁presentation']), (6, ['▁)']), (83, ['▁would']), (44, ['▁be']), (374, ['▁great']), (9, ['.']), (3, ['[SEP]'])]


 So the ids look good: the token `" ("` is now tokenized into one token which use the same id as the token `"("` before and "presentation" is no longer divided into 2 tokens 🚀