DeBERTa V3 Fast Tokenizer
Author: ikergarcia1996
Pull request: no
Labels: name: Good First Issue

# 🚀 Feature request

Fast Tokenizer for DeBERTA-V3 and mDeBERTa-V3 

## Motivation

DeBERTa V3 is an improved version of DeBERTa. With the V3 version, the authors also released a multilingual model "mDeBERTa-base" that outperforms XLM-R-base. However, DeBERTa V3 currently lacks a FastTokenizer implementation which makes it impossible to use with some of the example scripts (They require a FastTokenizer). 
DeBERTa-V1 and DeBERTa-V2 both have a FastTokenizer implementation, it would be great to have one for DeBERTa-V3.



Comments:
From my understanding DeBERTa-V3 has the same tokenizer as V2.
The problem is Transformers DeBERTa-V2 does not have a FastTokenizer implementation,
so we need a request for V2 FastTokenizer.

That would be a nice community contribution! I'll add the `Good First Issue` label, and happy to guide anyone from the community to add a DeBERTa v2 fast tokenizer with @SaulLu!

I've looked into it, but the only "problem" I see is the own SPMTokenizer implementation (for the slow tokenizer). It basically wraps spm, but does some own preprocessing steps...



Indeed, it would be great to have fast versions for the tokenizers of the models : 
- deberta-v2
- deberta-v3
- mdeberta-v3

As @stefan-it  raised, I think we are indeed missing some information to be able to build a rust version of the spm tokenizer used. I tried to have a quick look at the papers corresponding to each of the models (https://arxiv.org/abs/2006.03654 and https://arxiv.org/abs/2111.09543) and unfortunately, the tokenizer modeling is not explained in them. 

As far as I know, there is no way to retro-engineer the spm binaries, but maybe I'm wrong! So, at least, I think we need to know the command used by the authors to train their tokenizer, I saw that several people asked for it on issues on their [repo](https://github.com/microsoft/DeBERTa) ([issue 1](https://github.com/microsoft/DeBERTa/issues/82), [issue 2](https://github.com/microsoft/DeBERTa/issues/38)) but the answer is not in it. So, the only tracks I see are 1) to bounce on the existing issues to indicate that we would also be interested in this information or 2) to contact directly the authors. Indeed, if we know the command that was used to train this tokenizer, we should be able to assemble the right tokenizer components to have a fast tokenizer!

Moreover, concerning mdeberta-v3, in their paper they mention that :
> We denote the model as mDeBERTa-base. We use the same SentencePiece vocabulary as mT5 which has 250k tokens.

But unfortunately the binaries [spiece.model](https://huggingface.co/google/mt5-base/blob/main/spiece.model) for mT5 and [spm.model](https://huggingface.co/microsoft/mdeberta-v3-base/blob/main/spm.model) for mdeberta-v3 are not equal  

Gently removing the "Good First Issue" label while solving these issues. Also pinging the author @BigBird01 :)

There's also a vocab mismatch:

mDeBERTa: 251000
mT5: 250112



I think the most comfortable solution would be to use the T5 Fast Tokenizer (using the mDeBERTa vocab file) - but there are a lot of details to be checked:

Token mapping is different (t5):

https://github.com/huggingface/transformers/blob/824fd44fc314b0d13b1ef91e8122d35d14af5ad9/src/transformers/models/t5/tokenization_t5.py#L113-L116

DeBERTa:

https://github.com/huggingface/transformers/blob/824fd44fc314b0d13b1ef91e8122d35d14af5ad9/src/transformers/models/deberta_v2/tokenization_deberta_v2.py#L108-L116

:thinking: 

In DeBERTa tokenizer, we remapped [CLS]=>1, [PAD]=>0, [UNK]=>3, [SEP]=>2 while keep other pieces unchanged.

I checked [T5Converter](https://github.com/huggingface/transformers/blob/824fd44fc314b0d13b1ef91e8122d35d14af5ad9/src/transformers/convert_slow_tokenizer.py#:~:text=class%20T5Converter(SpmConverter)%3A), I think it should work by directly use T5Converter to convert deberta v2/v3 tokenizer to faster tokenizer, except for the post_processor part: 

tokenizer.post_processor = processors.TemplateProcessing(
            single="[CLS]:0 $A:0 [SEP]:0",
            pair="[CLS]:0 $A:0 [SEP]:0 $B:0 [SEP]:0",
            special_tokens=[
                ("[CLS]", self.original_tokenizer.convert_tokens_to_ids("[CLS]")),
                ("[SEP]", self.original_tokenizer.convert_tokens_to_ids("[SEP]")),
            ],
        )

Thanks!


Thank you so much for your super fast response @BigBird01 :heart_eyes: ! It's a great help to us!

It looks like all the information is there for this to be a good first issue :confetti_ball: !

So I'm putting the label back on and would be really happy to guide with a todo list and help the person who would like to take care of this new feature! :relaxed: 

Hi all, thanks for all the information provided, I have written a Converter Class for DeBERTav2 and testing it manually myself the tokenization looks correct. However I need guidance on how could I write up a DeBERTav2TokenizerFast class so I can add tests! Really appreciate any guidance, thank you!

> In DeBERTa tokenizer, we remapped [CLS]=>1, [PAD]=>0, [UNK]=>3, [SEP]=>2 while keep other pieces unchanged.
> 
> I checked [T5Converter](https://github.com/huggingface/transformers/blob/824fd44fc314b0d13b1ef91e8122d35d14af5ad9/src/transformers/convert_slow_tokenizer.py#:~:text=class%20T5Converter(SpmConverter)%3A), I think it should work by directly use T5Converter to convert deberta v2/v3 tokenizer to faster tokenizer, except for the post_processor part:
> 
> tokenizer.post_processor = processors.TemplateProcessing( single="[CLS]:0 $A:0 [SEP]:0", pair="[CLS]:0 $A:0 [SEP]:0 $B:0 [SEP]:0", special_tokens=[ ("[CLS]", self.original_tokenizer.convert_tokens_to_ids("[CLS]")), ("[SEP]", self.original_tokenizer.convert_tokens_to_ids("[SEP]")), ], )
> 
> Thanks!

Doing it the way you said I ran into this problem：

The class this function is called from is 'DebertaV2Tokenizer'.
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
Traceback (most recent call last):
  File "prepare_debertav3_data.py", line 89, in <module>
    tokenizer = DebertaV2Converter(tokenizer).converted()
  File "/search/odin/lida/.conda/envs/py36/lib/python3.6/site-packages/transformers/convert_slow_tokenizer.py", line 428, in __init__
    with open(self.original_tokenizer.vocab_file, "rb") as f:
AttributeError: 'DebertaV2Tokenizer' object has no attribute 'vocab_file'

A possible solution is mentioned here https://www.kaggle.com/nbroad/deberta-v2-3-fast-tokenizer

> A possible solution is mentioned here https://www.kaggle.com/nbroad/deberta-v2-3-fast-tokenizer

@kpriyanshu256 , I made that and I got it from here 😉 

Adding fast tokenizer is not needed. Because this model does not even work for the core task it is supposed to work - the masked language model [MASK].




> Adding fast tokenizer is not needed. Because this model does not even work for the core task it is supposed to work - the masked language model [MASK].

Am I right in understanding that the NER pipeline only returns character indexes for tagged spans when a fast tokenizer is used?  This could be one advantage of implementing.

> Adding fast tokenizer is not needed. Because this model does not even work for the core task it is supposed to work - the masked language model [MASK].

Even if the language model weights are not included, the model can still be used for fine-tuning for tasks such as text classification, QA, NER, etc. Since a lot (all?) of the Hugging Face scripts require FastTokenizers, the fast tokenizer is very much needed. 

It is very close to being merged to main: (see here https://github.com/huggingface/transformers/pull/15529)

@nbroad1881
I agree but also MLM task should be remove from the huggingface documentation or there should be warning that there are not weights for MLM task.

Are you sure these other tasks such as NER are going to be as good as in the PDF describing Deberta?









Edit: v2 was, v3 wasn't
One other note is that v2 and v3 were not trained with MLM

On Sat, Apr 2, 2022 at 16:05 Oxi84 ***@***.***> wrote:

> @nbroad1881 <https://github.com/nbroad1881>
> I agree but also MLM task should be remove from the huggingface
> documentation or there should be warning that there are not weights for MLM
> task.
>
> Are you sure these other tasks such as NER are going to be as good as in
> the PDF describing Deberta?
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/transformers/issues/14712#issuecomment-1086713611>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AF6TKFPLYWNPU2OBVLM5IRLVDCSA3ANCNFSM5JY6BIPA>
> .
> You are receiving this because you were mentioned.Message ID:
> ***@***.***>
>


The fast tokenizers are now merged to main through https://github.com/huggingface/transformers/pull/15529

why it is closed? the issue is not solved! It has errors when use return_offsets_mapping, could you open it?
I am using 4.18.0:
```
tokenizer = AutoTokenizer.from_pretrained("microsoft/deberta-v3-base", use_fast=True)
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
>>> tokenizer.encode_plus('what it is', return_offsets_mapping=True)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.7/dist-packages/transformers/tokenization_utils_base.py", line 2566, in encode_plus
    **kwargs,
  File "/usr/local/lib/python3.7/dist-packages/transformers/tokenization_utils.py", line 639, in _encode_plus
    "return_offset_mapping is not available when using Python tokenizers. "
NotImplementedError: return_offset_mapping is not available when using Python tokenizers. To use this feature, change your tokenizer to one deriving from transformers.PreTrainedTokenizerFast. More information on available tokenizers at https://github.com/huggingface/transformers/pull/2674
```

@world2vec please update your transformers version, i don't think the pip has been updated to > 4.19.0 yet
```
>>> import transformers
>>> transformers.__version__
'4.21.0.dev0'
>>> from transformers import AutoTokenizer
>>> tokenizer = AutoTokenizer.from_pretrained("microsoft/deberta-v3-base", use_fast=True)

Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
>>> tokenizer.encode_plus('what it is', return_offsets_mapping=True)
{'input_ids': [1, 339, 278, 269, 2], 'token_type_ids': [0, 0, 0, 0, 0], 'attention_mask': [1, 1, 1, 1, 1], 'offset_mapping': [(0, 0), (0, 4), (4, 7), (7, 10), (0, 0)]}```

It was added in 4.19 :hugs: 

https://github.com/huggingface/transformers/releases/tag/v4.19.0