TypeError loading tokenizer for gpssohi/distilbart-qgen-6-6
Author: bogolese
Pull request: no

Attempting to create a summary pipeline using "gpssohi/distilbart-qgen-6-6" as I get the message:

```
OSError: Can't load config for 'gpssohi/distilbart-qgen-6-6'. Make sure that:
- 'gpssohi/distilbart-qgen-6-6' is a correct model identifier listed on 'https://huggingface.co/models'
- or 'gpssohi/distilbart-qgen-6-6' is the correct path to a directory containing a config.json file
```
This despite the instructions on the model card:

```
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

tokenizer = AutoTokenizer.from_pretrained("gpssohi/distilbart-qgen-6-6")

model = AutoModelForSeq2SeqLM.from_pretrained("gpssohi/distilbart-qgen-6-6")
```

So I downloaded the model files locally and ran:

```
from transformers import BartTokenizer
tokenizer = BartTokenizer.from_pretrained("/pub/models/gpssohi/distilbart-qgen-6-6")
```

which produces the error:

```
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 3343, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-4-9811fff8faaa>", line 2, in <module>
    tokenizer = BartTokenizer.from_pretrained("/pub/models/gpssohi/distilbart-qgen-6-6")
  File "/usr/local/lib/python3.6/site-packages/transformers/tokenization_utils_base.py", line 1428, in from_pretrained
    return cls._from_pretrained(*inputs, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/transformers/tokenization_utils_base.py", line 1575, in _from_pretrained
    tokenizer = cls(*init_inputs, **init_kwargs)
  File "/usr/local/lib/python3.6/site-packages/transformers/tokenization_roberta.py", line 174, in __init__
    **kwargs,
  File "/usr/local/lib/python3.6/site-packages/transformers/tokenization_gpt2.py", line 169, in __init__
    super().__init__(bos_token=bos_token, eos_token=eos_token, unk_token=unk_token, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/transformers/tokenization_utils.py", line 116, in __init__
    super().__init__(**kwargs)
  File "/usr/local/lib/python3.6/site-packages/transformers/tokenization_utils_base.py", line 1314, in __init__
    super().__init__(**kwargs)
  File "/usr/local/lib/python3.6/site-packages/transformers/tokenization_utils_base.py", line 658, in __init__
    "special token {} has to be either str or AddedToken but got: {}".format(key, type(value))
TypeError: special token bos_token has to be either str or AddedToken but got: <class 'dict'>
```

I did some spelunking through the code and found that bos_token (and its siblings) are loaded via file tokenizer_config.json, which contains:

`{"unk_token": {"content": "<unk>", "single_word": false, "lstrip": false, "rstrip": false, "normalized": true, "__type": "AddedToken"}, "bos_token": {"content": "
", "single_word": false, "lstrip": false, "rstrip": false, "normalized": true, "__type": "AddedToken"}, "eos_token": {"content": "
", "single_word": false, "lstrip": false, "rstrip": false, "normalized": true, "__type": "AddedToken"}, "add_prefix_space": false, "errors": "replace", "sep_token": {"content": "
", "single_word": false, "lstrip": false, "rstrip": false, "normalized": true, "__type": "AddedToken"}, "cls_token": {"content": "
", "single_word": false, "lstrip": false, "rstrip": false, "normalized": true, "__type": "AddedToken"}, "pad_token": {"content": "<pad>", "single_word": false, "lstrip": false, "rstrip": false, "normalized": true, "__type": "AddedToken"}, "mask_token": {"content": "<mask>", "single_word": false, "lstrip": true, "rstrip": false, "normalized": true, "__type": "AddedToken"}, "model_max_length": 1024, "special_tokens_map_file": null, "name_or_path": "sshleifer/distilbart-cnn-6-6", "tokenizer_class": "BartTokenizer"}`

This is loaded via json.load, resulting in the value of each token being (yup!), a dictionary!  Now the value of the __type key for each token makes it seem like these are serialized AddedToken values, which, if properly reconstituted, would let this run without error.

FYI, from site-packages/torch/version.py:

```
version = ‘1.8.1+cu102’
debug = False
cuda = ‘10.2’
git_version = ‘56b43f4fec1f76953f15a627694d4bba34588969’
hip = None
```

Comments:
Hello! Was this a temporary error? Running your code sample seems to work for me:

```
In [1]: from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
   ...:
   ...: tokenizer = AutoTokenizer.from_pretrained("gpssohi/distilbart-qgen-6-6")
Downloading: 100%|███████████████████████████████████████████████████████████████████████████████████| 1.10k/1.10k [00:00<00:00, 471kB/s]
Downloading: 100%|████████████████████████████████████████████████████████████████████████████████████| 780k/780k [00:00<00:00, 1.53MB/s]
Downloading: 100%|████████████████████████████████████████████████████████████████████████████████████| 446k/446k [00:00<00:00, 1.03MB/s]
Downloading: 100%|██████████████████████████████████████████████████████████████████████████████████| 1.29M/1.29M [00:00<00:00, 2.28MB/s]
Downloading: 100%|███████████████████████████████████████████████████████████████████████████████████████| 772/772 [00:00<00:00, 209kB/s]

In [2]: model = AutoModelForSeq2SeqLM.from_pretrained("gpssohi/distilbart-qgen-6-6")
   ...:
Downloading: 100%|██████████████████████████████████████████████████████████████████████████████████| 1.83k/1.83k [00:00<00:00, 1.41MB/s]
Downloading:  35%|█████████████████████████████▊                                                      | 311M/877M [00:25<04:32, 2.18MB/s]
```

Thank you for responding!  I greatly appreciate it.

I'm afraid to say this is a permanent error (for me anyway).  :(

Perhaps this is a version issue?

From pip list:

```
torch                  1.8.1
transformers           3.3.1
```

Also:

```
transformers-cli env

Copy-and-paste the text below in your GitHub issue and FILL OUT the two last points.

- `transformers` version: 3.3.1
- Platform: Linux-3.10.0-1160.59.1.el7.x86_64-x86_64-with-centos-7.9.2009-Core
- Python version: 3.6.8
- PyTorch version (GPU?): 1.8.1+cu102 (False)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No
```

When I run in this environment (in the Python Console in PyCharm):

```
/usr/bin/python3.6 /pub/apps/pycharm-community-2020.3.4/plugins/python-ce/helpers/pydev/pydevconsole.py --mode=client --port=36152
import sys; print('Python %s on %s' % (sys.version, sys.platform))
sys.path.extend(['/pub/dev/ner-extract'])
Python 3.6.8 (default, Nov 16 2020, 16:55:22) 
Type 'copyright', 'credits' or 'license' for more information
IPython 7.16.1 -- An enhanced Interactive Python. Type '?' for help.
PyDev console: using IPython 7.16.1
Python 3.6.8 (default, Nov 16 2020, 16:55:22) 
[GCC 4.8.5 20150623 (Red Hat 4.8.5-44)] on linux
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
tokenizer = AutoTokenizer.from_pretrained("gpssohi/distilbart-qgen-6-6")
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/transformers/configuration_utils.py", line 359, in get_config_dict
    raise EnvironmentError
OSError
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 3343, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-2-511f5761aa2d>", line 2, in <module>
    tokenizer = AutoTokenizer.from_pretrained("gpssohi/distilbart-qgen-6-6")
  File "/usr/local/lib/python3.6/site-packages/transformers/tokenization_auto.py", line 216, in from_pretrained
    config = AutoConfig.from_pretrained(pretrained_model_name_or_path, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/transformers/configuration_auto.py", line 310, in from_pretrained
    config_dict, _ = PretrainedConfig.get_config_dict(pretrained_model_name_or_path, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/transformers/configuration_utils.py", line 368, in get_config_dict
    raise EnvironmentError(msg)
OSError: Can't load config for 'gpssohi/distilbart-qgen-6-6'. Make sure that:
- 'gpssohi/distilbart-qgen-6-6' is a correct model identifier listed on 'https://huggingface.co/models'
- or 'gpssohi/distilbart-qgen-6-6' is the correct path to a directory containing a config.json file
model = AutoModelForSeq2SeqLM.from_pretrained("gpssohi/distilbart-qgen-6-6")
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/transformers/configuration_utils.py", line 359, in get_config_dict
    raise EnvironmentError
OSError
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 3343, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-3-3fbaa62107f0>", line 1, in <module>
    model = AutoModelForSeq2SeqLM.from_pretrained("gpssohi/distilbart-qgen-6-6")
  File "/usr/local/lib/python3.6/site-packages/transformers/modeling_auto.py", line 1074, in from_pretrained
    pretrained_model_name_or_path, return_unused_kwargs=True, **kwargs
  File "/usr/local/lib/python3.6/site-packages/transformers/configuration_auto.py", line 310, in from_pretrained
    config_dict, _ = PretrainedConfig.get_config_dict(pretrained_model_name_or_path, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/transformers/configuration_utils.py", line 368, in get_config_dict
    raise EnvironmentError(msg)
OSError: Can't load config for 'gpssohi/distilbart-qgen-6-6'. Make sure that:
- 'gpssohi/distilbart-qgen-6-6' is a correct model identifier listed on 'https://huggingface.co/models'
- or 'gpssohi/distilbart-qgen-6-6' is the correct path to a directory containing a config.json file
```

So, I downloaded all torch related files from https://huggingface.co/gpssohi/distilbart-qgen-6-6/tree/main to local directory /pub/models/gpssohi/distilbart-qgen-6-6 and ran:

```
/usr/bin/python3.6 /pub/apps/pycharm-community-2020.3.4/plugins/python-ce/helpers/pydev/pydevconsole.py --mode=client --port=38897
import sys; print('Python %s on %s' % (sys.version, sys.platform))
sys.path.extend(['/pub/dev/ner-extract'])
Python 3.6.8 (default, Nov 16 2020, 16:55:22) 
Type 'copyright', 'credits' or 'license' for more information
IPython 7.16.1 -- An enhanced Interactive Python. Type '?' for help.
PyDev console: using IPython 7.16.1
Python 3.6.8 (default, Nov 16 2020, 16:55:22) 
[GCC 4.8.5 20150623 (Red Hat 4.8.5-44)] on linux
from transformers import AutoConfig, BartTokenizer
config = AutoConfig.from_pretrained("/pub/models/gpssohi/distilbart-qgen-6-6")
tokenizer = BartTokenizer.from_pretrained("/pub/models/gpssohi/distilbart-qgen-6-6")
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 3343, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-2-55dbdfaca761>", line 5, in <module>
    tokenizer = BartTokenizer.from_pretrained("/pub/models/gpssohi/distilbart-qgen-6-6")
  File "/usr/local/lib/python3.6/site-packages/transformers/tokenization_utils_base.py", line 1428, in from_pretrained
    return cls._from_pretrained(*inputs, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/transformers/tokenization_utils_base.py", line 1575, in _from_pretrained
    tokenizer = cls(*init_inputs, **init_kwargs)
  File "/usr/local/lib/python3.6/site-packages/transformers/tokenization_roberta.py", line 174, in __init__
    **kwargs,
  File "/usr/local/lib/python3.6/site-packages/transformers/tokenization_gpt2.py", line 169, in __init__
    super().__init__(bos_token=bos_token, eos_token=eos_token, unk_token=unk_token, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/transformers/tokenization_utils.py", line 116, in __init__
    super().__init__(**kwargs)
  File "/usr/local/lib/python3.6/site-packages/transformers/tokenization_utils_base.py", line 1314, in __init__
    super().__init__(**kwargs)
  File "/usr/local/lib/python3.6/site-packages/transformers/tokenization_utils_base.py", line 658, in __init__
    "special token {} has to be either str or AddedToken but got: {}".format(key, type(value))
TypeError: special token bos_token has to be either str or AddedToken but got: <class 'dict'>
```

As I said earlier, it seems unable to load the list of AddedToken from the content of the tokenizer_config.json.

So this *does* appear to be a version issue.  I had to upgrade transformers to 4.1.0 (4.0.0 did not work), which also required upgrading sentence-transformers and it upgraded tokenizers to 0.9.4.