Cannot load saved tokenizer using AutoTokenizer
Author: trias702
Pull request: no
Labels: name: wontfix

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: 3.4.0
- Platform: Win10 x64 (1607 Build 14393.3866)
- Python version: 3.6.10
- PyTorch version (GPU?): 1.5.1
- Tensorflow version (GPU?): None
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.
 
 albert, bert, GPT2, XLM: @LysandreJik 
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Speed and Memory Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 Translation: @sshleifer
 Summarization: @sshleifer
 TextGeneration: @TevenLeScao 
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @TevenLeScao
 blenderbot: @mariamabarham
 Bart: @sshleifer
 Marian: @sshleifer
 T5: @patrickvonplaten
 Longformer/Reformer: @patrickvonplaten
 TransfoXL/XLNet: @TevenLeScao 
 examples/seq2seq: @sshleifer
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 -->
@mfuntowicz
## Information

It appears that you can save a tokenizer to disk in a model agnostic way, but you cannot load it back in a model agnostic way. Is this a bug or by design?

## To reproduce

Steps to reproduce the behavior:

```
tokenizer = AutoTokenizer.from_pretrained('roberta-base')
tokenizer.save_pretrained('TEST/tokenizer')

tokenizer = AutoTokenizer.from_pretrained('TEST/tokenizer')
# ERROR
```

The error you get is because the config argument is None, which means AutoTokenizer calls AutoConfig.from_pretrained, which utilises file_utils.CONFIG_NAME, however tokenizer.save_pretrained uses tokenization_utils_base.TOKENIZER_CONFIG_FILE instead, so they're not compatible with one another.


## Expected behavior

I would assume that calling AutoTokenizer.from_pretrained would be able to load and instantiate the correct model tokenizer without the user having to directly import the model tokenizer class first (e.g. RobertaTokenizer.from_pretrained). This would help a lot in moving to a model agnostic way of handling tokenizers, which I feel is the goal of the AutoTokenizer class. The fact that it can't load a tokenizer from disk seems to be a bug, unless there is a different way of doing this?


Comments:
Hello! Indeed, I wouldn't say this is a bug but more of a limitation of the `AutoTokenizer` class that has to rely on the model configuration in order to guess which tokenizer is affiliated with the model. Since you're not interacting with the configuration in the configuration anywhere here, and, therefore, are not saving the model configuration in `TEST/tokenizer`, the AutoTokenizer cannot guess from which tokenizer to load.

One way to go around this limitation is to either specify the configuration when loading the tokenizer for the second time:
```py
from transformers import AutoTokenizer, AutoConfig

tokenizer = AutoTokenizer.from_pretrained('roberta-base')
tokenizer.save_pretrained('TEST/tokenizer')

tokenizer = AutoTokenizer.from_pretrained('TEST/tokenizer', config=AutoConfig.from_pretrained("roberta-base"))
```
Another way would be to save the configuration in the initial folder:
```py
from transformers import AutoTokenizer, AutoConfig

tokenizer = AutoTokenizer.from_pretrained('roberta-base')
config = AutoConfig.from_pretrained('roberta-base')

tokenizer.save_pretrained('TEST/tokenizer')
config.save_pretrained('TEST/tokenizer')

tokenizer = AutoTokenizer.from_pretrained('TEST/tokenizer')
```

In any case, the documentation about this should be improved.

Thank you for that reply, I very much appreciate it!

What about the following, would this work also?

```
tokenizer = AutoTokenizer.from_pretrained('roberta-base')
# make changes to tokenizer, for example add custom tokens
tokenizer.save_pretrained('TEST/tokenizer')

tokenizer = AutoTokenizer.from_pretrained('roberta-base')
tokenizer = tokenizer.from_pretrained('TEST/tokenizer')
```

If you do it this way, when you call the last line of the code, will you restore any changes you previously made to the tokenizer?

Finally, for what's it worth, I do believe that the way the library is doing it now is wrong, from a design philosophy perspective. The Tokenizers should be able to stand completely apart from their models, as they are their own classes, with their own configs and config format. You shouldn't need the Model Config in order to save down and restore a tokenizer, because you can do it entirely without the model if you call the direct model tokenizer class:

```
tokenizer = RobertaTokenizer.from_pretrained('roberta-base')
tokenizer.save_pretrained('TEST/tokenizer')

tokenizer = RobertaTokenizer.from_pretrained('TEST/tokenizer')
# WORKS
```

So it really should not make any difference if you execute the same design pattern, but from a model-agnostic way, as in my original example:

```
tokenizer = AutoTokenizer.from_pretrained('roberta-base')
tokenizer.save_pretrained('TEST/tokenizer')

tokenizer = AutoTokenizer.from_pretrained('TEST/tokenizer')
# ERROR, but really should work
```

The AutoTokenizer class should just be about Tokenizer, and should not be calling AutoConfig (which is for models). Basically, you need an AutoTokenConfig class instead, which decouples the two. Calling `save_pretrained` on a Tokenizer (any tokenizer) should save all the information about it (including it's model-class, for example RobertaTokenizer) such that you can then load it from disk using AutoTokenizer, and the AutoTokenizer would be smart enough to check the files on disk, read some JSON info, and say "Ah yes, this should be a RobertaTokenizer" and then return to you a RobertaTokenizer object, even though you called AutoTokenizer.from_pretrained. In fact, as it stands now, this information about tokenizer type is already being written to disk, it's just not being read back by the AutoTokenizer. If you created an AutoTokenizerConfig class with its own tokenizer-specific config reading-from-disk methods, then you could easily accomplish this.

The reason this would be a powerful design pattern to have is you could make complex language modelling pipelines across different scripts and the tokenizer would only need to be class specified once, at the topmost script.

For example, say you have a script which preprocesses a custom corpus for downstream language modelling, and it does this using Shelve, creating compressed records ready to be read by a downstream collator class. But in the same directory, it also saves down the (custom) tokenizer used, let's say a modified RobertaTokenizer.

The downstream script would not need to know anything about RobertaTokenizer, all it does is read in the Shelve records, and loads the tokenizer using AutoTokenizer.from_pretrained, and then just runs what it needs to run, and hands its results to yet another downstream process, and then that process also just loads the tokenizer using AutoTokenizer.from_pretrained, and doesn't need to know anything about what type of tokenizer it is, because it just uses the PretrainedTokenizer base class methods.

So the only script that ever knew about RobertaTokenizer was the very first one, and it saved it using save_pretrained, and then all of the downstream worker scripts just load that tokenizer using AutoTokenizer.from_pretrained. This allows all the downstream scripts to be model-agnostic, and not need to know about RobertaTokenizer at all, meaning they could work with any PretrainedTokenizer at all.

This is a very efficient pipeline that makes full use of the abstract base classes like PretrainedTokenizer. Otherwise you need each of your downstream scripts to be model-specific, because they need to be told to use RobertaTokenizer instead of BertTokenizer instead of GPT2Tokenizer, etc.

The only thing that's missing to make this all work is for AutoTokenizer.from_pretrained to work in the manner which I have original tried to make it work.

While we aim for tokenizers and models to be pairs and not standalone classes, I do agree it would be better from a user perspective to put the tokenizer's class directly in the `tokenizer_config.json`, so as to work in the use-case that you mention here. We could add a flag to the configuration, similar to the `architecture` that we have in the model configuration.

Thoughts @julien-c, @thomwolf ?

Yes, sounds good to me indeed

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.