[WIP] Fast tokenizer for debertaV2
Author: alcinos
Pull request: yes

# What does this PR do?

Implements a fast tokenizer for deberta v2. Loosely based on #11387

Fixes #11529
Fixes #14712 

This is a draft as there are some failing tests (not super clear to me why atm, will have to investigate)


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?

@LysandreJik


Comments:
I noticed that while I was working on my PR, another was submitted for the same purpose: #14923. 

Hey @alcinos ,

thanks for adding it!

I'm currently running comparisons between slow and fast tokenizer. Here are some mismatches between fast and slow.

I just run tokenization tests on `README.md` and `README_zh-hans.md` from official Transformers library, using this script:

```python
import sys

from transformers import DebertaV2Tokenizer, DebertaV2TokenizerFast

model_name = "microsoft/deberta-v2-xlarge"

slow_tokenizer = DebertaV2Tokenizer.from_pretrained(model_name)
fast_tokenizer = DebertaV2TokenizerFast.from_pretrained(model_name)

filename = sys.argv[1]

with open(filename, "rt") as f_p:
    for line in f_p:
        line = line.rstrip()

        if not line:
            continue

        slow_tokens = slow_tokenizer.tokenize(line)
        fast_tokens = fast_tokenizer.tokenize(line)

        if slow_tokens != fast_tokens:
            print("Tokenization mismatch:", line)
            print("Slow tokens:", slow_tokens)
            print("Fast tokens:", fast_tokens)
```

Here are some mismatches:

Original input: `* 🖼️ Images, for tasks like image classification, object detection, and segmentation.`
Slow tokens: `['▁*', '▁', '[UNK]', '️', '▁Images', ',', '▁for', '▁tasks', '▁like', '▁image', '▁classification', ',', '▁object', '▁detection', ',', '▁and', '▁segmentation', '.']`
Fast tokens: `['▁*', '▁', '🖼', '️', '▁Images', ',', '▁for', '▁tasks', '▁like', '▁image', '▁classification', ',', '▁object', '▁detection', ',', '▁and', '▁segmentation', '.']`

Another example on `README_zh-hans.md`:

Original input: `- 对教学和实践友好且低门槛`
Slow tokens: `['▁-', '▁', '对', '教', '学', '和', '实', '践', '友', '好', '且', '低', '门', '[UNK]']`
Fast tokens: `['▁-', '▁', '对', '教', '学', '和', '实', '践', '友', '好', '且', '低', '门', '槛']`

The original DeBERTa tokenizer outputs the same tokens as the slow tokenizer.

@stefan-it Thanks for looking into this and providing the testcases.

It seems that the issues you are reporting are all related to unknown tokens? I don’t know the rust implementation well enough, is there any reason why the fast tokenizer would not respect the vocabulary?

Hey @alcinos I'm currently trying to figure it out :)

Good news: when using `encode`, there's no mismatch between slow and fast tokenizer.

For slow tokenizer, this is happening here:

https://github.com/huggingface/transformers/blob/501307b58bdc2db1b6a25271a3f60975130f1c6c/src/transformers/models/deberta_v2/tokenization_deberta_v2.py#L326-L330

Also when using normal T5 (Slow and fast) there are no UNKs when using the `tokenize` function (but `encode` shows that those subtokens are UNKs) so this is DeBERTa-specific.

@alcinos  I think the issue has something regarding the tokenize function inherited from `PreTrainedTokenizerFast` 

For this line: `* 🖼️ Images, for tasks like image classification, object detection, and segmentation.` 

the `tokenize` function called`encode_plus` returns a Dict[List] converted from BatchEncoding. For both slow and fast tokenizers `encode_plus` return ```{'input_ids': [943, 250, 3, 28596, 7654, 6, 14, 2930, 72, 812, 8692, 6, 2328, 5563, 6, 7, 27235], 'token_type_ids': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]}```

https://github.com/huggingface/transformers/blob/501307b58bdc2db1b6a25271a3f60975130f1c6c/src/transformers/tokenization_utils_fast.py#L316-L317
 but the .tokens() at the end of the line doesn't return the [UNK] token but rather the token itself which causes the discrepancy here. 

May not be entirely correct - but I was able to fix the discrepancy in the testing script @stefan-it provided by overriding the tokenize function and adding this method in Debertav2TokenizerFast class
```
    def tokenize(self, text: str, pair: Optional[str] = None, add_special_tokens: bool = False, **kwargs) -> List[str]:
        enc = self.encode_plus(text=text, text_pair=pair, add_special_tokens=add_special_tokens, **kwargs)
        return self.convert_ids_to_tokens(enc['input_ids'])
```

But the other tests are still failing and I'm not sure what's causing the issue and need to investigate. 


Thanks @stefan-it and @mingboiz for looking into the tokenization issue. If I summarize the findings so far:

- Slow tokenizer replaces unknown tokens with "[UNK]" while fast tokenizer doesn’t
- This behavior seems specific to Deberta, as T5’s tokenizers don’t replace with "[UNK]"
- After encoding, the results are the same for slow and fast, meaning that the issue is probably minor
- @mingboiz found a way to have the fast tokenizer spit out the "[UNK]".

I’m not sure what the expected behavior should be, nor whether we should be concerned about this in the first place. Input from someone from HF would be appreciated :) (ping @SaulLu )


Aside from that, I pushed some fixes, more tests are passing.
Some feedback for the HF team on the issue I ran into:
In one of the common tests, the code looks for a "do_lower_case" attribute variable: 
https://github.com/huggingface/transformers/blob/10fd4fa1a6ca08b6bba5fed2db666a55239d717c/tests/test_tokenization_common.py#L626-L627
This is problematic in my opinion since:
- I didn’t see any mention that this attribute variable is required in the documentation I’ve come across (though I may have missed it)
- The code silently fails if the attribute is not present
- This argument itself is not used anywhere in the DebertaV2TokenizeFast class, hence I was not naturally inclined to add it as an attribute.

I would suggest one of the following change to make this more dev friendly:
- Make it a hard requirement that any tokenizer class must have this attribute, document it, and remove the silent fail in case it’s not found
- Additionally, IMHO this would be better suited as a an overridable getter method rather than a direct access to a private attribute


More tests are failing: one most likely has the same root cause as the issue raised by @stefan-it. The others seem to be failing because in some code paths the vocab_file is None, but it’s not clear to me why that happens, any help on that appreciated.

@alcinos I can't figure out a solution yet but the tests that are failing because of the missing vocab file which I think it's because in all of them legacy_format=False is being selected 
https://github.com/huggingface/transformers/blob/f80775df2b418716acce76d56826ed59183869b6/tests/test_tokenization_common.py#L3513
which only saves using the Rust Tokenizer these files without the `spm.model` vocab file: 

```
tokenizer_config.json',
special_tokens_map.json',
tokenizer.json'
```
this code chunk will run instead without using the save_vocabulary function:

https://github.com/huggingface/transformers/blob/f80775df2b418716acce76d56826ed59183869b6/src/transformers/tokenization_utils_fast.py#L578-L583

Debertav2Tokenizer didn't have this issue because its backend SPMTokenizer class provided its own `save_pretrained` method to save the `spm.model`, but I can't figure out why the AlbertTokenizerFast tests could work and passes all tests when the same tests fails here -  which I think these are the only remaining failing tests:

- test_saving_tokenizer_trainer
- test_training_new_tokenizer_with_special_tokens_change
- test_training_new_tokenizer

I have noted the ping! I'll come back to you as soon as possible on this subject because the choice to be made here is not obvious: you have highlighted that the `tokenize` method of `DebertaV2Tokenizer` does not behave in the same way as all the tokenizer methods of the fast tokenizers. At the moment it would make more sense to me to modify the `DebertaV2Tokenizer`'s `tokenize` method, but in general we don't really like to introduce backward incompatibility, so I'll need to discuss it with the maintainers :slightly_smiling_face:.

@alcinos and @mingboiz , while investigating the `test_saving_tokenizer_trainer` test further, I noticed that the variable `VOCAB_FILES_NAMES`  did not specify the `"tokenizer_file"` value (and we need it for the fast version of the tokenizer). Moreover, for the test to fully succeed, the following lines must also be removed from the `tokenization_deberta_v2_fast.py` file.

```
        if not os.path.isfile(vocab_file):
            raise ValueError(
                f"Can't find a vocabulary file at path '{vocab_file}'. To load the vocabulary from a Google pretrained "
                "model use `tokenizer = AutoTokenizer.from_pretrained(PRETRAINED_MODEL_NAME)`"
            )
```
With these 2 changes, the test now pass :smile: !

I have opened a [PR](https://github.com/alcinos/transformers/pull/1/files) here to show you the changes that should be made to solve these problems. Feel free to merge it if you agree with it. :slightly_smiling_face: 

@alcinos could you please have a look at the https://github.com/alcinos/transformers/pull/1 PR - I think it is ready then :hugs: 

Hi @alcinos, thank you very much for your work, the addition seems to be near the end! Please let me know if you need help with any of it!

Finished in PR #15529

Thanks all again for the contribution :hugs: