

TITLE
ValueError: An instance of tokenizer class BioGptTokenizer cannot be converted  in a Fast tokenizer instance. No converter was found.

COMMENTS
7

REACTIONS
+1: 0
-1: 0
laugh: 0
hooray: 0
heart: 0
rocket: 0
eyes: 0

BODY
### System Info

ValueError: An instance of tokenizer class BioGptTokenizer cannot be converted  in a Fast tokenizer instance. No converter was found.

I am using microsoft/biogpt for token classification ner task script(https://github.com/huggingface/transformers/blob/main/examples/pytorch/token-classification/run_ner.py ) which is having slow tokenizer and the journey been so far is this

1. Got an error in the above mentioned script as this ("This example script only works for models that have a fast tokenizer.")
2. And then went for second option that is mentioned which is to use old script In which I got an Runtime error and I reported the issue and got as an answer that I need to use new run_ner.py
3. I got an option to convert slow tokenizer to fast but now I am getting this error 

"ValueError: An instance of tokenizer class BioGptTokenizer cannot be converted 
in a Fast tokenizer instance. No converter was found. Currently available 
slow->fast convertors: ['AlbertTokenizer', 'BartTokenizer', 'BarthezTokenizer', 
'BertTokenizer', 'BigBirdTokenizer', 'BlenderbotTokenizer', 
'CamembertTokenizer', 'CLIPTokenizer', 'CodeGenTokenizer', 'ConvBertTokenizer', 
'DebertaTokenizer', 'DebertaV2Tokenizer', 'DistilBertTokenizer', 
'DPRReaderTokenizer', 'DPRQuestionEncoderTokenizer', 
'DPRContextEncoderTokenizer', 'ElectraTokenizer', 'FNetTokenizer', 
'FunnelTokenizer', 'GPT2Tokenizer', 'HerbertTokenizer', 'LayoutLMTokenizer', 
'LayoutLMv2Tokenizer', 'LayoutLMv3Tokenizer', 'LayoutXLMTokenizer', 
'LongformerTokenizer', 'LEDTokenizer', 'LxmertTokenizer', 'MarkupLMTokenizer', 
'MBartTokenizer', 'MBart50Tokenizer', 'MPNetTokenizer', 'MobileBertTokenizer', 
'MvpTokenizer', 'NllbTokenizer', 'OpenAIGPTTokenizer', 'PegasusTokenizer', 
'RealmTokenizer', 'ReformerTokenizer', 'RemBertTokenizer', 'RetriBertTokenizer',
'RobertaTokenizer', 'RoFormerTokenizer', 'SqueezeBertTokenizer', 'T5Tokenizer', 
'WhisperTokenizer', 'XLMRobertaTokenizer', 'XLNetTokenizer', 
'SplinterTokenizer', 'XGLMTokenizer', 'LlamaTokenizer']"

It is my request to team to add BioGptTokenizer in the list.

### Who can help?

_No response_

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

%cd /content/transformers/examples/pytorch/token-classification
!python run_ner.py \
  --tokenizer_name microsoft/biogpt\
  --model_name_or_path microsoft/biogpt\
  --train_file /content/TRAIN.json \
  --validation_file  /content/DEV.json \
  --test_file  /content/DEV.json \
  --output_dir $checkpoint_dir \
  --per_device_train_batch_size 4 \
  --per_device_eval_batch_size 4 \
  --do_train \
  --do_eval \
  --do_predict \
  --num_train_epochs 4\
  --evaluation_strategy epoch\
  --task_name ner\
  --overwrite_output_dir True\
  --save_strategy epoch\
  --ignore_mismatched_sizes=True

### Expected behavior

Successfully train after the conversion