[pretrained] model classes aren't checking the arch of the pretrained model it loads
Author: stas00
Pull request: no
Labels: name: Good First Issue

While comparing different models trained on xsum (most of which are Bart) I made a mistake and passed "google/pegasus-xsum" to `BartForConditionalGeneration`  

```
BartForConditionalGeneration.from_pretrained("google/pegasus-xsum")
```

I got:

```
Some weights of the model checkpoint at google/pegasus-xsum were not used when initializing BartForConditionalGeneration: ['model.encoder.layer_norm.weight', 'model.encoder.layer_norm.bias', 'model.decoder.layer_norm.weight', 'model.decoder.layer_norm.bias']
- This IS expected if you are initializing BartForConditionalGeneration from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing BartForConditionalGeneration from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Some weights of BartForConditionalGeneration were not initialized from the model checkpoint at google/pegasus-xsum and are newly initialized: ['model.encoder.embed_positions.weight', 'model.encoder.layernorm_embedding.weight', 'model.encoder.layernorm_embedding.bias', 'model.decoder.embed_positions.weight', 'model.decoder.layernorm_embedding.weight', 'model.decoder.layernorm_embedding.bias']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
Traceback (most recent call last):
  File "./bart-summarize2.py", line 8, in <module>
    tokenizer = BartTokenizer.from_pretrained(mname)
  File "/mnt/nvme1/code/huggingface/transformers-master/src/transformers/tokenization_utils_base.py", line 1788, in from_pretrained
    return cls._from_pretrained(
  File "/mnt/nvme1/code/huggingface/transformers-master/src/transformers/tokenization_utils_base.py", line 1860, in _from_pretrained
    tokenizer = cls(*init_inputs, **init_kwargs)
  File "/mnt/nvme1/code/huggingface/transformers-master/src/transformers/models/roberta/tokenization_roberta.py", line 159, in __init__
    super().__init__(
  File "/mnt/nvme1/code/huggingface/transformers-master/src/transformers/models/gpt2/tokenization_gpt2.py", line 179, in __init__
    with open(vocab_file, encoding="utf-8") as vocab_handle:
TypeError: expected str, bytes or os.PathLike object, not NoneType
```

Any reason why the model class doesn't check that it's being fed a wrong architecture? It could detect that and give a corresponding error message, rather than spitting random errors like above? I was pretty sure it was a bug in pegasus model until I noticed that pegasus != Bart.

Thanks.

@LysandreJik 


Comments:
Ah indeed, that's a good request ! There's no reason, we could definitely raise a warning when loading the weights by checking the model type in the configuration against the arch's model type. Do you want to open a PR?

Why a warning and not an assert? If the code throws a totally unrelated long backtrace how would a user know to search for an earlier warnings?

Do you see a use-case where someone may need to load mismatching arch for the given model?

After thinking about it, you're right that an error would be better. I can't think of use-cases where that would affect someone's workflow negatively.

> Do you want to open a PR?

I could, but realistically it might not happen soon. But since it's not a complicated task perhaps asking the community to help? I guess it'd be as simple as:

1. read the config of the downloaded model as soon as the config got downloaded
2. compare `config.arch` with model's arch
3. assert if mismatch

Hi @LysandreJik Does someone work on that ? I'd like to make my first contribution to the project

Hi @ankh6, feel free to work on it! The issue is not reserved until a PR is opened with some progress made towards solving the issue.

And when you solve it, one test can be:
```
python -c 'from transformers import PegasusForConditionalGeneration; PegasusForConditionalGeneration.from_pretrained("patrickvonplaten/t5-tiny-random")'
```
but this one doesn't crash, just spits a lot of warnings. 

This one does crash:
```
python -c 'from transformers import BartForConditionalGeneration; BartForConditionalGeneration.from_pretrained("prajjwal1/bert-tiny")'
```

So it'd be a better candidate to go into the test suite. 

We want a tiny model so that it runs the test fast.

@LysandreJik  If I understand correctly we should check that the input is in the PRETRAINED_VOCAB_FILES_MAP object (for this issue). Should the assertion occur when we call is_torch_available method, i.e. in src/transformers/models/gpt2/__init__.py, ? 

As soon as you retrieved the config file and you know which model's class is used, so that you have the 2 things to compare.

It definitely shouldn't happen in the specific model files, but inside the common library.

Most likely there should be one check inside one of the super-classes for model's `from_pretrained` and the same for the tokenizer. Since either may have this conflict.

Hi, 
I've made some progress on this issue. Think I've fixed it for initiating models. 
To show if my approach is fine shall I submit a PR? 

I've essentially added an assert statement in the `from_pretrained` method in the `PretrainedConfig` class.  

That sounds about right, and yes PR please - thank you!

Added a pull request #10586