DebertaTokenizer Rework closes #10258
Author: cronoik
Pull request: yes

# What does this PR do?

Fixes #10258

@BigBird01 Please upload these [files](https://drive.google.com/drive/folders/1gH5EMABR94iHO7SCb_AdNGCOOIdSloxh?usp=sharing) to your deberta repositories.
@huggingface: Please don't merge before @BigBird01 has uploaded the files to his repository.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [X] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [X] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

- tokenizers:  @LysandreJik, @BigBird01 


Comments:
This is great @cronoik! I just tested it and reproduced identical results between the previous version and this one. Fantastic!

Could you add an integration tests for the DeBERTa tokenizer to ensure the implementations don't diverge? You can just copy paste the [test for ALBERT tokenizers](https://github.com/huggingface/transformers/blob/master/tests/test_tokenization_albert.py#L106) and change the values to DeBERTa.

If you don't have time, let us know and we'll take care of it.

@LysandreJik 
I will give it a try.

Hi,
a new test is now added for the debertatokenizer. I stuck more to the robertatokenizer test since the debertatokenizer is basically a robertatokenizer.

I can not interpret the failed check. I assume it is because the model hub is missing the new files? @LysandreJik could you please have a look?

I am trying out this PR with the following:

```
tokenizer = DebertaTokenizer.from_pretrained("microsoft/deberta-base")
target_tokenized = tokenizer.tokenize("Some test text")
```
but I see the following error (`TypeError: expected str, bytes or os.PathLike object, not NoneType`):
```
    def __init__(
        self,
        vocab_file,
        merges_file,
        errors="replace",
        unk_token="
",
        bos_token="
",
        eos_token="
",
        add_prefix_space=False,
        **kwargs
    ):
        bos_token = AddedToken(bos_token, lstrip=False, rstrip=False) if isinstance(bos_token, str) else bos_token
        eos_token = AddedToken(eos_token, lstrip=False, rstrip=False) if isinstance(eos_token, str) else eos_token
        unk_token = AddedToken(unk_token, lstrip=False, rstrip=False) if isinstance(unk_token, str) else unk_token
        super().__init__(
            errors=errors,
            unk_token=unk_token,
            bos_token=bos_token,
            eos_token=eos_token,
            add_prefix_space=add_prefix_space,
            **kwargs,
        )

>       with open(vocab_file, encoding="utf-8") as vocab_handle:
E       TypeError: expected str, bytes or os.PathLike object, not NoneType

../../../../jeswan_transformers/src/transformers/models/gpt2/tokenization_gpt2.py:179: TypeError
```
@LysandreJik potentially related to @cronoik's error above 

@jeswan that will not work since the required files are not uploaded to the model hub yet. You can download them from the [link](https://drive.google.com/drive/folders/1gH5EMABR94iHO7SCb_AdNGCOOIdSloxh?usp=sharing) and load the tokenizer from local.

@cronoik thanks for your effort. I just uploaded the files to the model repository and left one comment to the changes. 

@BigBird01 Thank you for the review. I had only tested single sentences and completely ignored the sentence pairs before pushing.
@LysandreJik Can you please help me with the test error? 

@sgugger Thanks for the review. I have pushed/accepted your change requests.

Thanks for your efforts @cronoik!