odd whitespace handling with imported sentencepiece models
Author: tlby
Pull request: no
Labels: name: Core: Tokenization

## Environment info

- `transformers` version: 4.7.0
- Platform: Linux-4.15.0-143-generic-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.9
- PyTorch version (GPU?): 1.9.0+cu102 (False)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no

### Who can help
@LysandreJik Although my example uses ReformerTokenizer, I think this problem is present in several of the model architectures using sentencepiece tokenizers.

## Information

Model I am using (Bert, XLNet ...): Reformer

The problem arises when using:
* [x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

```
#!/usr/bin/env python3

import sentencepiece as spm
import transformers as tr

src = (
    'Lorem Ipsum dolor sit amet, consectetur adipiscing elit, sed do',
    'eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut',
    'enim ad minim veniam, quis nostrud exercitation ullamco laboris',
    'nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in',
    'reprehenderit in voluptate velit esse cillum dolore eu fugiat',
    'nulla pariatur.  Excepteur sint occaecat cupidatat non proident,',
    'sunt in culpa qui officia deserunt mollit anim id est laborum.',
)

spm.SentencePieceTrainer.train(
    sentence_iterator=iter(src),
    model_prefix='test',
    vocab_size=96,
    treat_whitespace_as_suffix=True,
    user_defined_symbols=['<pad>', '<mask>'],
    minloglevel=1,
)

def show(label, toks):
    print('%14s %2d: %s' % (label, len(toks), toks))

text = 'Lo<mask>m Ipsum'

tok = spm.SentencePieceProcessor(model_file='test.model')
show('sentencepiece', tok.encode(text, out_type=str))

tok = tr.models.reformer.ReformerTokenizerFast('test.model',
    mask_token='<mask>',
    pad_token='<pad>')
show('transformers', tok.tokenize(text))

tok.save_pretrained('test')

tr.models.reformer.ReformerConfig().save_pretrained('test')
tok = tr.AutoTokenizer.from_pretrained('test')
show('AutoTokenizer', tok.tokenize(text))
```
is giving
```
 sentencepiece  9: ['L', 'o', '<mask>', 'm▁', 'I', 'p', 's', 'um', '▁']
  transformers 10: ['▁', 'L', 'o', '<mask>', 'm', '▁', 'I', 'p', 's', 'um']
 AutoTokenizer 11: ['▁', 'L', 'o', '<mask>', '▁', 'm', '▁', 'I', 'p', 's', 'um']
```

## Expected behavior

I believe the tokenization of input text should be more consistent.  I think these variations are cropping up between my attempts to pretrain a language model and then later finetune the saved model, resulting in model accuracy problems.

The use of `treat_whitespace_as_suffix=True` in `sentencepiece` makes this problem worse, but using a sentencepiece model without this flag still shows the `AutoTokenizer.from_pretrained()` created tokenizer inserting whitespace that was not present in the source text.  I haven't been able to track down where this is coming from or how to avoid it.

Comments:
`spm.SentencePieceTrainer` and `ReformerTokenizerFast` are not the same tokenizers, so it's not unusual that each of them outputs different results.
However, I'm not sure how the two tokenizers are different. It's because of the lack of my knowledge.

Regarding the difference between `ReformerTokenizerFast` and `AutoTokenizer`, I discovered something.
One of the easiest ways to make the two tokenizers output the same results is to remove `mask_token='<mask>'` and `test` directory where previous config files exist (if there is `test` folder).

Another way is to remove `special_tokens_map.json` and `tokenizer_config.json`  (after `save_pretrained`) that are unnecessary files when using fast tokenizer.

I don't know what is the cause of this problem, but I guess there are conflicts between configurations of fast tokenizer and tokenizer.




This may have two underlying causes, one perhaps a serialization issue between `.save_pretrained()` and `AutoTokenizer.from_pretrained()`, and a separate issue related to different behavior between `PreTrainedTokenizer` and `PreTrainedTokenizerFast`.

Here is perhaps a clearer example of the variations:
```python
#!/usr/bin/env python3

import sentencepiece as spm
import transformers as tr

src = (
    'Lorem Ipsum dolor sit amet, consectetur adipiscing elit, sed do',
    'eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut',
    'enim ad minim veniam, quis nostrud exercitation ullamco laboris',
    'nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in',
    'reprehenderit in voluptate velit esse cillum dolore eu fugiat',
    'nulla pariatur.  Excepteur sint occaecat cupidatat non proident,',
    'sunt in culpa qui officia deserunt mollit anim id est laborum.',
)

spm.SentencePieceTrainer.train(
    sentence_iterator=iter(src),
    model_prefix='test',
    vocab_size=96,
    treat_whitespace_as_suffix=True,
    user_defined_symbols=['<pad>', '<mask>'],
    minloglevel=1,
)

def show(label, toks):
    print('%8s %2d: %s' % (label, len(toks), toks))

text = 'Lo<mask>m Ipsum'

cfg = tr.T5Config()

tok = tr.T5Tokenizer('test.model', mask_token='<mask>', pad_token='<pad>')
show('tr.slow', tok.tokenize(text))
cfg.save_pretrained('test_slow')
tok.save_pretrained('test_slow')

tok = tr.AutoTokenizer.from_pretrained('test_slow', use_fast=False)
show('at.slow', tok.tokenize(text))

tok = tr.T5TokenizerFast('test.model', mask_token='<mask>', pad_token='<pad>')
show('tr.fast', tok.tokenize(text))
cfg.save_pretrained('test_fast')
tok.save_pretrained('test_fast')

tok = tr.AutoTokenizer.from_pretrained('test_fast')
show('at.fast', tok.tokenize(text))
```
giving
```
 tr.slow  9: ['L', 'o', '<mask>', 'm▁', 'I', 'p', 's', 'um', '▁']
 at.slow 10: ['L', 'o', '▁', '<mask>', 'm▁', 'I', 'p', 's', 'um', '▁']
 tr.fast 10: ['▁', 'L', 'o', '<mask>', 'm', '▁', 'I', 'p', 's', 'um']
 at.fast 11: ['▁', 'L', 'o', '<mask>', '▁', 'm', '▁', 'I', 'p', 's', 'um']
```
The first one is consistent with `sentencepiece` directly, which is not surprising because these tokenizers use `spm.SentencePieceProcessor()` to encode.

@europeanplaice It looks like you're making headway on the serialization part, which is great.  I can file a separate ticket for the differences between the `PreTrainedTokenizer` and `PreTrainedTokenizerFast` subclasses if that part turns out unrelated.

@tlby
Thank you for giving another example.
It helped me a lot.
For now, I'm not sure whether the two underlying causes are unrelated.

The following is about the serialization issue.

I've found that `tr.T5Tokenizer` or `ReformerTokenizer` don't expect `mask_token`. So this attribute is taken as an element of `kwargs`. (fast tokenizer may also be the same.)

Refer to 
https://huggingface.co/transformers/model_doc/t5.html#t5tokenizer
https://huggingface.co/transformers/model_doc/reformer.html#reformertokenizer

I guess that when first initializing the tokenizer, it ignores `mask_token`, but when you recreate it by `from_pretrained`, it treats `mask_token` as a special token, then it tokenizes text differently.

In the first initialization, the tokenizer uses the default setting.
So it doesn't consider `mask_token` even if a user passes it as an argument. It gives the text to sentencepiece without any preprocessing, and `sentencepiece` recognizes `<mask>` as a mask token.
However, the tokenizer also writes `<mask>` as a special token in the config JSON.

When recreating the tokenizer by `from_pretrained`, it processes the tokens defined in the config JSON.
Before passing the text to sentencepiece, it splits the text to ['Lo', '`<mask>`', 'm Ipsum'] and sentencepiece tokenizes the elements except for `<mask>` then combines.

When I removed  `mask_token='<mask>'` as below
```python
tok = tr.T5Tokenizer('test.model', pad_token='<pad>')
tok = tr.T5TokenizerFast('test.model', pad_token='<pad>')
```
then results were
```
 tr.slow  9: ['L', 'o', '<mask>', 'm▁', 'I', 'p', 's', 'um', '▁']
 at.slow  9: ['L', 'o', '<mask>', 'm▁', 'I', 'p', 's', 'um', '▁']
 tr.fast 10: ['▁', 'L', 'o', '<mask>', 'm', '▁', 'I', 'p', 's', 'um']
 at.fast 10: ['▁', 'L', 'o', '<mask>', 'm', '▁', 'I', 'p', 's', 'um']
```

It would explain the difference between `tr.slow` and `at.slow`.
It also would explain the difference between `tr.fast` and `at.fast`.



When I do not specify a mask_token to the `PreTrainedTokenizer`, trying to use it in `examples/pytorch/language-modeling/run_mlm.py` gives errors.
```
[ERROR|tokenization_utils_base.py:1017] 2021-06-26 22:40:50,070 >> Using mask_token, but it is not set yet.
Traceback (most recent call last):
  File "run_mlm.py", line 504, in <module>
    main()
  File "run_mlm.py", line 438, in main
    pad_to_multiple_of=8 if pad_to_multiple_of_8 else None,
  File "<string>", line 6, in __init__
  File "/usr/local/lib/python3.6/dist-packages/transformers/data/data_collator.py", line 335, in __post_init__
    "This tokenizer does not have a mask token which is necessary for masked language modeling. "
ValueError: This tokenizer does not have a mask token which is necessary for masked language modeling. You should pass `mlm=False` to train on causal language modeling instead.
```
I don't have a standalone example of this, but if you need one I can work one out.

Pinging @n1t0 for advice

@LysandreJik @n1t0
Thank you for checking this conversation.

My situation is somewhat more sensitive to whitespace than most western languages because I am hoping to do language modeling with HTML source code.  In a sample such as `<a href="page.html">the page</a>` the insertion of whitespace after an attribute name `<a href ="page.html">the page</a>` takes us outside the realm of samples in the training set, which is why this problem is significant.

Rather than trying to recycle one of the existing `sentencepiece` based tokenizers, I worked out [my own](/tlby/rnd-html/blob/e65f246cece32b55f1a49e76f5bcad8dfc077839/mytok.py) PreTrainedTokenizer subclass I hope will be compatible with various transformer architectures.  So far it is working nicely for BERT in `run_mlm.py`.  The `AutoTokenizer.from_pretrained()` instance is behaving consistently and since I don't implement a PreTrainedTokenizerFast I don't have problems with it getting upgraded and changing tokenization behavior.

If you can confirm this approach isn't an obvious anti-pattern, then that might be enough to consider my issue resolved.

Tokenizers are generally engineered for specific use-cases and don't always adapt to different domains. The tokenizers here didn't fit your use-cases so you went and implemented one yourself so that it best fits your particular problem - I believe this is the best way to handle it and ensure it behaves consistently across your tasks.

I took a look at your `PreTrainedTokenizer` and I think you have everything covered! If ever you have some time available, it would be very helpful for us to know if you ran into issues implementing that subclass, or general ideas of how we could make it easier for you to implement custom tokenizers such as this one. For example, adding an option to register new tokenizers (such as the proposition in https://github.com/huggingface/transformers/issues/10256) to the `AutoTokenizer` mapping would probably have come in handy.

If you think of anything else, please feel free to open an issue with proposals, even if very high-level, so that we may improve the API. Thank you!

> If ever you have some time available, it would be very helpful for us to know if you ran into issues implementing that subclass

As with most mildly complicated inheritance APIs I had a couple of false starts trying to write something clean and minimalist first, then something heavy that got way out of hand.  Once I tracked down a working example that seemed closest to what I was trying to do, retooling and cleaning progressed rapidly.


Probably the biggest technical hurdle was assuming I wanted a "Fast" tokenizer and trying too hard to trick the `tokenizers` library into doing what I needed, which was almost, but not quite possible.

That's interesting feedback, thank you. We'll keep this in mind when working on improving the tokenizer API.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.