ValueError: An instance of tokenizer class BioGptTokenizer cannot be converted  in a Fast tokenizer instance. No converter was found.
Author: TekeshwarHirwani
Pull request: no
Labels: name: wontfix

### System Info

ValueError: An instance of tokenizer class BioGptTokenizer cannot be converted  in a Fast tokenizer instance. No converter was found.

I am using microsoft/biogpt for token classification ner task script(https://github.com/huggingface/transformers/blob/main/examples/pytorch/token-classification/run_ner.py ) which is having slow tokenizer and the journey been so far is this

1. Got an error in the above mentioned script as this ("This example script only works for models that have a fast tokenizer.")
2. And then went for second option that is mentioned which is to use old script In which I got an Runtime error and I reported the issue and got as an answer that I need to use new run_ner.py
3. I got an option to convert slow tokenizer to fast but now I am getting this error 

"ValueError: An instance of tokenizer class BioGptTokenizer cannot be converted 
in a Fast tokenizer instance. No converter was found. Currently available 
slow->fast convertors: ['AlbertTokenizer', 'BartTokenizer', 'BarthezTokenizer', 
'BertTokenizer', 'BigBirdTokenizer', 'BlenderbotTokenizer', 
'CamembertTokenizer', 'CLIPTokenizer', 'CodeGenTokenizer', 'ConvBertTokenizer', 
'DebertaTokenizer', 'DebertaV2Tokenizer', 'DistilBertTokenizer', 
'DPRReaderTokenizer', 'DPRQuestionEncoderTokenizer', 
'DPRContextEncoderTokenizer', 'ElectraTokenizer', 'FNetTokenizer', 
'FunnelTokenizer', 'GPT2Tokenizer', 'HerbertTokenizer', 'LayoutLMTokenizer', 
'LayoutLMv2Tokenizer', 'LayoutLMv3Tokenizer', 'LayoutXLMTokenizer', 
'LongformerTokenizer', 'LEDTokenizer', 'LxmertTokenizer', 'MarkupLMTokenizer', 
'MBartTokenizer', 'MBart50Tokenizer', 'MPNetTokenizer', 'MobileBertTokenizer', 
'MvpTokenizer', 'NllbTokenizer', 'OpenAIGPTTokenizer', 'PegasusTokenizer', 
'RealmTokenizer', 'ReformerTokenizer', 'RemBertTokenizer', 'RetriBertTokenizer',
'RobertaTokenizer', 'RoFormerTokenizer', 'SqueezeBertTokenizer', 'T5Tokenizer', 
'WhisperTokenizer', 'XLMRobertaTokenizer', 'XLNetTokenizer', 
'SplinterTokenizer', 'XGLMTokenizer', 'LlamaTokenizer']"

It is my request to team to add BioGptTokenizer in the list.

### Who can help?

_No response_

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

%cd /content/transformers/examples/pytorch/token-classification
!python run_ner.py \
  --tokenizer_name microsoft/biogpt\
  --model_name_or_path microsoft/biogpt\
  --train_file /content/TRAIN.json \
  --validation_file  /content/DEV.json \
  --test_file  /content/DEV.json \
  --output_dir $checkpoint_dir \
  --per_device_train_batch_size 4 \
  --per_device_eval_batch_size 4 \
  --do_train \
  --do_eval \
  --do_predict \
  --num_train_epochs 4\
  --evaluation_strategy epoch\
  --task_name ner\
  --overwrite_output_dir True\
  --save_strategy epoch\
  --ignore_mismatched_sizes=True

### Expected behavior

Successfully train after the conversion

Comments:
Hi @TekeshwarHirwani, thanks for raising this issue. 

Indeed, there doesn't exist a convert yet for this tokenizer. Would you like to add it? You can find examples of [converters here](https://github.com/huggingface/transformers/blob/main/src/transformers/convert_slow_tokenizer.py), and it will need to be added to the [SLOW_TO_FAST_CONVERTERS mapping](https://github.com/huggingface/transformers/blob/f4e4b4d0e2dc248433e808594f7595292037d891/src/transformers/convert_slow_tokenizer.py#L1230).

cc @Rocketknight1 @ArthurZucker  

Also other models based on `moses` don't have a fast tokenizer version (`xlm`, `fsmt`, `flaubert` etc). It's probably because moses is already fast enough and `tokenizers` library is not really made for ruled base tokenization. Correct me if I am wrong @Narsil 

Seems very correct. The reasons to skip `moses` are vague to me but indeed I'm not sure we should go down that path :).


@TekeshwarHirwani A similar context was raised here: https://github.com/huggingface/transformers/pull/17254#issuecomment-1150669010 

You may first create a slow-to-fast-converter that is similar to PhobertConverter/BertweetConverter from https://github.com/datquocnguyen/transformers/blob/main/src/transformers/convert_slow_tokenizer.py 
Then you could create a BiogptTokenizerFast in the same manner to as PhobertTokenizerFast/BertweetTokenizerFast from https://github.com/datquocnguyen/transformers/blob/main/src/transformers/models/bertweet/tokenization_bertweet_fast.py 

See more details [here](https://github.com/huggingface/transformers/pull/17254/files).

I am not really sure why `moses` was mentioned here @ArthurZucker @Narsil @amyeroberts 
The reason why you'd have to **hack** the `tokenizers` to have a fast variant of such slow tokenizers for FlauBERT or BioGPT is that [many subwords appearing in the "merges" file do not appear in the "vocab" file as in CTRL, FlauBERT, BioGPT, PhoBERT and BERTweet and the like (i.e. slow tokenizers would convert those subwords into unkn_id during encoding), thus it is impossible to develop a fast tokenizer variant using documented approaches while keeping the same tokenization strategy](https://github.com/huggingface/transformers/pull/17254#issuecomment-1150669010).

Thanks for the link.
I'm confused in the thread you linked you say that fast tokenizers are possible: https://github.com/huggingface/transformers/pull/17254#issuecomment-1130248921 and I see one here: https://huggingface.co/vinai/bertweet-covid19-base-uncased/blob/main/tokenizer.json

This lead me to check BiotGPT **does** use moses: https://github.com/huggingface/transformers/blob/main/src/transformers/models/biogpt/tokenization_biogpt.py#L156-L162
Flaubert **uses it too**: https://github.com/huggingface/transformers/blob/main/src/transformers/models/flaubert/tokenization_flaubert.py#L312-L318
While CTRL **doesn't**: https://github.com/huggingface/transformers/blob/main/src/transformers/models/ctrl/tokenization_ctrl.py
And phobert indeed **doesn't** : https://github.com/huggingface/transformers/blob/main/src/transformers/models/phobert/tokenization_phobert.py

So phobert might be doable, but I'm not sure it's related to BioGPT


@Narsil Please be aware of the difference between "subword" tokenization vs. "word" tokenization. 

All the `tokenization_{model_name}.py` files you mentioned use "bpe" for `subword` tokenization, e.g. [https://github.com/huggingface/transformers/blob/2ab75add4b30c2fc44a8bf575156d448d9ed87a7/src/transformers/models/biogpt/tokenization_biogpt.py#L170](https://github.com/huggingface/transformers/blob/2ab75add4b30c2fc44a8bf575156d448d9ed87a7/src/transformers/models/biogpt/tokenization_biogpt.py#L170)

BiotGPT, Flaubert, CTRL, PhoBERT and BERTweet all have "merges" and "vocab" files for BPE-based subword tokenization (e.g. see https://huggingface.co/microsoft/biogpt/tree/main). 

For BiotGPT and Flaubert, `mosestokenizer` is just a `word` tokenizer/normalizer, which can be used as an external preprocess w.r.t. a fast `subword` tokenization variant (likewise,  to perform Vietnamese word segmentation before using PhobertTokenizerFast, or to perform Tweet normalization before using BertweetTokenizerFast). 

PS: https://huggingface.co/vinai/bertweet-covid19-base-uncased/blob/main/tokenizer.json is just a saved output of the  [convert_slow_tokenizer.py](https://github.com/datquocnguyen/transformers/blob/main/src/transformers/convert_slow_tokenizer.py) that takes "merges" and "vocab" files as input. 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.