Wrong isinstance for AutoTokenizers with unspecified tokenizer_class
Author: leogao2
Pull request: no

## Environment info

<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
- `transformers` version: 4.5.0
- Platform: Linux-5.8.0-59-generic-x86_64-with-glibc2.29
- Python version: 3.8.5
- PyTorch version (GPU?): 1.8.1+cu111 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no

### Who can help

@LysandreJik

## Information

When using a model like https://huggingface.co/bigscience/tr3e-1B3-c4-checkpoints/tree/global_step117000/global_step117000 (since subfolder doesn't work this needs to be cloned), which is a GPT2 model using the T5 tokenizer, and with tokenizer_class=null in the config, instantiating an AutoTokenizer after copying T5 tokenizer files into the directory causes isinstance to show it as a GPT2Tokenizer instead.

## To reproduce

Steps to reproduce the behavior:

1. clone the model and checkout the branch
2. copy T5 tokenizer files into model directory
3. call `AutoTokenizer.from_pretrained` using that directory
4. look at `isinstance(tok, transformers.GPT2TokenizerFast)` (returns true) and `isinstance(tok, transformers.T5TokenizerFast)` (returns false)

I think what it's doing is using the tokenizer_class field to decide what class to pretend to be, and silently defaulting to the model type default if it's null. This has created some problems with some code I have that checks what tokenizer is being used.


Comments:
Hi @leogao2, thanks for opening an issue!

You're correct in your analysis. It first searches for `tokenizer_class`, and if not found, defaults to the tokenizer that comes with the model.

The `tokenizer_class` parameter is here for exactly that purpose: to have the `AutoTokenizer` understand that it shouldn't load the default tokenizer, but that it should instead load the tokenizer overridden by the user.

Would it be possible to put the `T5Tokenizer` value in the `tokenizer_class` of the model's configuration file?

> Hi @leogao2, thanks for opening an issue!
> 
> You're correct in your analysis. It first searches for `tokenizer_class`, and if not found, defaults to the tokenizer that comes with the model.

Should the search pattern be instead:

1. `config.json`'s `tokenizer_class`
2. check tokenizer files in the folder of the model
3. fallback to model's default tokenizer

> The `tokenizer_class` parameter is here for exactly that purpose: to have the `AutoTokenizer` understand that it shouldn't load the default tokenizer, but that it should instead load the tokenizer overridden by the user.
> 
> Would it be possible to put the `T5Tokenizer` value in the `tokenizer_class` of the model's configuration file?

We can. We just need to make the Megatron to HF conversion script to become aware of the tokenizer used in training.

We are just trying to figure out whether having the correct tokenizer files in the model's folder should be enough or not.

Either way works for us.


That's an interesting question! I think it isn't straightforward. Minus a few features that don't exist in the slow tokenizers, we'd like to have identical behavior between the slow and the fast tokenizers. For slow tokenizers, a lot of different tokenizers actually use the same files: the GPT-2 and RoBERTa tokenizers use the same underlying algorithm, and use the same files, so which one would you load given the files? The same can be said for BERT, DistilBERT, SqueezeBERT, etc.

For the fast tokenizers, it isn't straightforward either. The fast tokenizer only requires a single file: the `tokenizer.json` file, which contains the configuration of all components that make up the tokenizer, as well as its vocabulary. However, it isn't aware of what class it should load itself in, as it comes from the `tokenizers` library which doesn't have the notion of a `GPT2Tokenizer` or a `T5Tokenizer`.

When you load a `tokenizer.json` file in a tokenizer class, you're essentially creating the tokenizer from everything contained in the `tokenizer.json` file, and the class adds a few sensible defaults, usually linked to the special tokens (unknown, padding, etc).

You can try this out yourself by loading the `tokenizer.json` file of the `t5-small` checkpoint in both the `GPT2TokenizerFast` and `T5TokenizerFast`:

```py
>>> from transformers import T5TokenizerFast, GPT2TokenizerFast

>>> t5 = T5TokenizerFast.from_pretrained("<path_to_folder_containing_file>")
file <path_to_folder_containing_file>/config.json not found

>>> gpt2 = GPT2TokenizerFast.from_pretrained("<path_to_folder_containing_file>")
file <path_to_folder_containing_file>/config.json not found
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
```

If you encode with the two tokenizers, you should see the same result, as it's the same algorithm behind the scenes:

```py
>>> t5("Hey")
{'input_ids': [9459, 1], 'attention_mask': [1, 1]}

>>> gpt2("Hey")
{'input_ids': [9459, 1], 'attention_mask': [1, 1]}
```

However, if you check the defaults of the tokenizer, these should be different:

```py
>>> gpt2.unk_token
'
'

>>> t5.unk_token
'<unk>'
```

---

For the issue you're facing, I would suggest putting the expected class of your tokenizer in the `config.json` or in the `tokenizer_config.json`, which is the configuration of the tokenizer. Both configurations can set this attribute, and the `AutoTokenizer` will start first by checking the `tokenizer_config.json` file for the `tokenizer_class` attribute, before continuing with the `config.json` file, to finally default to the tokenizer linked to the model specified in the `config.json`.

Erratum: the ability to infer the tokenizer class from the `tokenizer_config.json` was introduced in version v4.8.0. I'm seeing in your issue description that you're using `v4.5.0`, so I would recommend sticking to `config.json` if you're planning on staying on that version.

Wow, perhaps this monumental study should be saved in the docs, as it's quite the unravelling. Thank you for that, Lysandre.

So based on your recommendation it sounds like **the conversion script should populate  `tokenizer_class` in the `config.json`.** I will add that.

I think it's safe to close this now. 

@leogao2, if you feel there is something else to discuss here please don't hesitate to reopen.