`from_pretrained` doesn't validate its kwargs
Author: stas00
Pull request: no

splitting off from https://github.com/huggingface/transformers/issues/13895

```
AutoModel.from_pretrained(mname, reviison=ckpt) # dyslexic typist
```
Throws no exception and it definitely should, since this could be unnoticed for a long time and meanwhile the user gets incorrect behavior.

Expected behavior:

There should be an assert if a user passes an argument that is not expected by the function.

@LysandreJik, @sgugger 

Comments:
I believe this is an issue with a specific model class:
```
from transformers import AutoModel

model = AutoModel.from_pretrained("bert-base-cased", reviison="main")
```

gives me

```
TypeError: __init__() got an unexpected keyword argument 'reviison'
```

Which checkpoint are you using?

Digging more, I think you're using a GPT-like model, and this is caused by [this line](https://github.com/huggingface/transformers/blob/5be59a364961a8e2fc986f1276cba977db87512a/src/transformers/models/gpt2/modeling_gpt2.py#L456) or the equivalent in other GPT model classes. Not sure what it achieves but it definitely causes the bug you are seeing.

**EDIT** Digging even more, the subclasses are actually not passing any kwargs, so I don't get the issue with GPT checkpoints. Do you have a checkpoint I can use to reproduce the issue?

> Which checkpoint are you using?

gpt2 indeed.

> Do you have a checkpoint I can use to reproduce the issue?

https://huggingface.co/bigscience/gpt2-1b3-en

I don't seem to have access to that model, as it gives me an 404 error. `"gpt2"` uses the same class, `GPT2LMHeadModel`, and does not produce the bug on master. It also looks like that model config was not saved using the `from_pretrained` method since it's not formatted properly, I don't know if that has something to do with the issue at hand.

Could you upload a smaller, public model that reproduces the bug ?

I thought you had a god mode on the hub, please re-try now?

the model was generated by the conversion script that runs https://github.com/huggingface/transformers/blob/master/src/transformers/models/megatron_gpt2/convert_megatron_gpt2_checkpoint.py

if it generates an invalid config, we should fix that script then.

I was missing the `use_atuh_token=True` to get the download going, but the following sample does not give me the bug:
```
from transformers import AutoModel

model = AutoModel.from_pretrained("bigscience/gpt2-1b3-en", reviison="main", use_auth_token=True)
```

gives me the error:

```
TypeError: __init__() got an unexpected keyword argument 'reviison'
```

indeed, let me try with the original model that was reported to have this issue.

Also you mentioned invalid config please see my updated comment above.

This [line](https://github.com/huggingface/transformers/blob/5be59a364961a8e2fc986f1276cba977db87512a/src/transformers/models/megatron_gpt2/convert_megatron_gpt2_checkpoint.py#L303) in the script does not save the config in the same format as the others. I don't know why they don't use the method of the config to properly save it.

Ha, the typo in your comment was a perfect use case:

```
python -c "from transformers import AutoTokenizer, AutoModel; mname = 'bigscience/gpt2-1b3-en'; \
AutoModel.from_pretrained(mname, use_auht_token=True)"
```

I suppose the validation happens much later in the game. The above should assert, instead it fails to access the model.

And my apologies for not posting a reproducible example from the get going - my bad! I was trying to report on someone else's behalf.


> This line in the script does not save the config in the same format as the others. I don't know why they don't use the method of the config to properly save it.

I will see what I can do about fixing that. 

It's probably a really old script.

> The above should assert, instead it fails to access the model.

I'm not sure if this what you meant, but the above cannot return an error it received a wrong kwarg as there is an error coming before we get to the end of using all the kwargs. We can't validate the kwargs at that level since we don't even know which model class to use, so we don't know which kwargs it will accept. So sadly there is nothing we can do about misspelled `use_auth_token=True` or other arguments that result in the urls of the model being wrong.

So here we have a bit of a chicken and egg problem, the args that are needed to get to the hub aren't validated until after the files are **successfully** fetched. I understand.

So it'd be the same issue with misspelled `revision` - it'd ignore it, and give the user the `main` branch instead. And then fail if there is actually a model under `main`. It was failing for us in a different way since we didn't have anything under `main` (different repo: `bigscience/tr3d-1B3-oscar-checkpoints`). All checkpoints were in branches. That explains why it was failing then and not reporting a misspelled kwargs.

That's quite the conundrum for the 2 kwargs then: `revision` and `use_auth_token` (not sure if there are other critical ones).


I suppose our use case is probably rare and it's probably ok to leave it as is. I think the only issue is `use_auth_token`

It'd probably  help the user if the hub were to report 403 (access denied), rather than:
```
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://huggingface.co/bigscience/gpt2-1b3-en/resolve/main/config.json
```
then the user is likely to review their config and discover they are either missing `use_auth_token` or it is misspelled.

Moreover, if the client side got 403 - it could have suggested to the user to pass `use_auth_token=True`.

on the server side, 403 rather than 404 leaks info because you can "discover" repo names for instance. That's why 404 is commonly used (on GitHub, etc)

Understood. Thank you for this insight, Julien.


This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.