broken models on the hub
Author: stas00
Pull request: no

Go to https://huggingface.co/sshleifer/distill-mbart-en-ro-12-6 click on "use in transformers", copy-n-paste and nope can't use this in `transformers`:

```
python -c 'from transformers import AutoTokenizer; AutoTokenizer.from_pretrained("sshleifer/distill-mbart-en-ro-12-6")'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/mnt/nvme1/code/huggingface/transformers-master/src/transformers/models/auto/tokenization_auto.py", line 410, in from_pretrained
    return tokenizer_class_fast.from_pretrained(pretrained_model_name_or_path, *inputs, **kwargs)
  File "/mnt/nvme1/code/huggingface/transformers-master/src/transformers/tokenization_utils_base.py", line 1704, in from_pretrained
    return cls._from_pretrained(
  File "/mnt/nvme1/code/huggingface/transformers-master/src/transformers/tokenization_utils_base.py", line 1717, in _from_pretrained
    slow_tokenizer = (cls.slow_tokenizer_class)._from_pretrained(
  File "/mnt/nvme1/code/huggingface/transformers-master/src/transformers/tokenization_utils_base.py", line 1776, in _from_pretrained
    tokenizer = cls(*init_inputs, **init_kwargs)
  File "/mnt/nvme1/code/huggingface/transformers-master/src/transformers/models/roberta/tokenization_roberta.py", line 159, in __init__
    super().__init__(
  File "/mnt/nvme1/code/huggingface/transformers-master/src/transformers/models/gpt2/tokenization_gpt2.py", line 179, in __init__
    with open(vocab_file, encoding="utf-8") as vocab_handle:
TypeError: expected str, bytes or os.PathLike object, not NoneType
```

this is with the latest master.

These for example I tested to work fine:

- `sshleifer/distill-mbart-en-ro-12-4`
- `sshleifer/distill-mbart-en-ro-12-9`


Perhaps we need a sort of CI that goes over the public models, validates that `run in transformers` code succeeds and sends an alert if it doesn't? We have no idea how many other models are broken on the hub right now.


Comments:
I fixed this model (it had `bart` as a `model_type` instead of `mbart` in `config.json`). 

But the point of this issue is that perhaps we could add to a todo list to run a cronjob that will validate the models and tokenizers? 

@theo-m, is this something that would fit with the project you're currently working on, since you will have to run these things anyway for each model? Just asking - not piling it up on you.

In fact if I understand the idea correctly this will be a requirement for your things to work, right?

We would not be aiming for 100% coverage, but yes, getting a sense of what's runnable on the hub would be awesome. 

Maybe a once-a-week CI job, as I expect the run to be terribly long/expensive? cc infra people @julien-c @n1t0 

I think we'll hook something into the git-push-event driven ML analytics system we've been talking about internally

That's a medium-term goal though

It's a different thing though: on-push would ensure integrity on upload, which is indeed needed, but a recurrent job would enable us to detect regression in what the lib can support and give estimates on what is actually runnable.

The lib can change after the upload was made and the model/tokenizer stop working. We have seen this before with older models.

I think @theo-m you're saying the same thing. 

@theo-m, btw the low hanging fruit would be to just validate that the listed in "use in transformers" instructions indeed work. i.e. we just load the model and tokenizer and do nothing with it if it works and do something with it if it doesn't. 

Note that this is not necessarily a low hanging fruit (depending on your definition of a low hanging fruit 😂) given that:
- we have 7,000+ models whose total weights represent multiple TBs of data
- they change over time

the lowest hanging fruit is loading all that can be associated to a pipeline and run a single example in the associated pipeline, the results of this are stronger than just loading

and yes it sure is a big big job, but it's the best we can do in order to build a good understanding of what is runnable on the hub - _in fine_ for non hf we won't be able to do much, but we can't give guarantees to code we don't manage.

I meant that just loading a model / tokenizer is cheaper/faster/requires almost 0 extra code to write - hence low-hanging fruit.

I hear you that the hub is huge, a little bit at a time. It would have been the same code to validate 10 models or 7K models if there is no urgency to complete it fast, it just would take much much longer to complete.

> * they change over time

That was exactly my point, they and the codebase too, so it's not enough to check it once, even if we track when it was changed and when it was validated last.


This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.