Default code snippet for shared model doesn't work because of different model and tokenizer types
Author: obsh
Pull request: no
Labels: name: wontfix

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): GPT2 with AlbertTokenizer

Language I am using the model on (English, Chinese ...): Ukrainian

The problem arises when using:
* [x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

When I share GPT2 model trained with AlbertTokenizer on huggingface.co the default code snippet doesn't work. The reason is that AutoTokenizer detects tokenizer type by model type.

## To reproduce

Steps to reproduce the behavior:

1. Go to https://huggingface.co/Tereveni-AI/gpt2-124M-uk-fiction
2. Copy code from default snippet `How to use this model directly from the 🤗/transformers library`
3. Try to execute snippet code

```
>>> from transformers import AutoTokenizer, AutoModel
>>> tokenizer = AutoTokenizer.from_pretrained("Tereveni-AI/gpt2-124M-uk-fiction")
Downloading: 100%|████████████████████████████████████████████████████████████████████████████| 645/645 [00:00<00:00, 89.4kB/s]
Downloading: 100%|████████████████████████████████████████████████████████████████████████████| 121/121 [00:00<00:00, 18.8kB/s]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/oleksandrbushkovskyi/Projects/transformers-ua/venv/lib/python3.7/site-packages/transformers/tokenization_auto.py", line 200, in from_pretrained
    return tokenizer_class_py.from_pretrained(pretrained_model_name_or_path, *inputs, **kwargs)
  File "/Users/oleksandrbushkovskyi/Projects/transformers-ua/venv/lib/python3.7/site-packages/transformers/tokenization_utils.py", line 898, in from_pretrained
    return cls._from_pretrained(*inputs, **kwargs)
  File "/Users/oleksandrbushkovskyi/Projects/transformers-ua/venv/lib/python3.7/site-packages/transformers/tokenization_utils.py", line 1051, in _from_pretrained
    tokenizer = cls(*init_inputs, **init_kwargs)
  File "/Users/oleksandrbushkovskyi/Projects/transformers-ua/venv/lib/python3.7/site-packages/transformers/tokenization_gpt2.py", line 151, in __init__
    with open(vocab_file, encoding="utf-8") as vocab_handle:
TypeError: expected str, bytes or os.PathLike object, not NoneType
```

## Expected behavior

I would expect `AutoTokenizer` to detect `spiece.model` and initialize `AlbertTokenizer`.
Even better if `AutoTokenizer` read tokenizer type explicitly defined in `tokenizer_config.json`.

## Environment info
- `transformers` version: 2.9.0
- Platform: Darwin-19.4.0-x86_64-i386-64bit
- Python version: 3.7.7
- PyTorch version (GPU?): 1.5.0 (False)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no


Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


I have the exact same issue. When I share GPT2 model trained with AlbertTokenizer on huggingface.co the default code snippet doesn't work. The reason is that AutoTokenizer detects tokenizer type by model type. Is there any workaround?

Hi @obsh and @khashei – yes you need to add a `tokenizer_class` attribute to your model's config.json:

> tokenizer_class (str, optional) – The name of the associated tokenizer class to use (if none is set, will use the tokenizer associated to the model by default).

https://huggingface.co/transformers/main_classes/configuration.html

cc @patrickvonplaten