AutoModel.from_config loads random parameter values.
Author: gkutiel
Pull request: no
Labels: name: Documentation

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): Bert

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)
Model parameters are (apparently) random initialized when using `AutoModel.from_config`.

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. `git clone https://github.com/gkutiel/transformers-bug`
2. `cd transformers-bug`
3. `pipenv shell`
4. `pipenv install`
5. `python main.py`

```python
from transformers import (
    AutoModel,
    AutoConfig,
)

pretrained = 'bert-base-uncased'

model_from_pretrained = AutoModel.from_pretrained(pretrained)
model_from_config = AutoModel.from_config(AutoConfig.from_pretrained(pretrained))

model_from_pretrained_params = list(model_from_pretrained.parameters())
model_from_config_params = list(model_from_config.parameters())

assert len(model_from_pretrained_params) == len(model_from_config_params)

model_from_pretrained_first_param = model_from_pretrained_params[0][0][0]
model_from_config_first_param = model_from_config_params[0][0][0]

assert model_from_pretrained_first_param == model_from_config_first_param, (
    f'{model_from_pretrained_first_param} != {model_from_config_first_param}'
)
```
## Expected behavior
An assertion error should not happen.

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: 2.10.0
- Platform: MacOS
- Python version:3.6
- PyTorch version (GPU?):
- Tensorflow version (GPU?):
- Using GPU in script?:
- Using distributed or parallel set-up in script?:


Comments:
This is expected behaviour, but I understand your confusion.

```python
model_from_pretrained = AutoModel.from_pretrained(pretrained)
```

This actually loads the pretrained weights. It looks up the mapping and locations of the config file and the weights, and loads both.

```python
model_from_config = AutoModel.from_config(AutoConfig.from_pretrained(pretrained))
```

Here, the pretrained weights are never requested. You request the pretrained _config_ (basically the pretraining settings for the architecture), and (randomly) initialise an AutoModel given that config - but the weights are never requested and, thus, never loaded. 

This means that both initialised models will have the same architecture, the same config, but different weights. The former has pretrained weights, the latter is randomly initialised.

I think that what you expected or wanted is actually this, which will load pretrained weights and taking into account a pretrained config (however, this is practically the same as the first option):

```python
model_from_config = AutoModel.from_pretrained(pretrained, config=AutoConfig.from_pretrained(pretrained))
```

Hope that helps.

Thank you very much for the fast response. 
I think that the documentation is not clear enough about this difference, especially when there are pre-trained models such as `bert-base-uncased` and `bert-base-cased`, and there is the `AutoModelForPreTraining` class (that now I'm not sure what is for).




If I understand correctly, your confusion lies in "well I called `.from_pretrained` so I would expect the model to have pretrained weights". However, the distinction is that if you run .from_pretrained on Auto**Config** you are not loading weights but you are loading a pre-existing config file. Loading pre-existing weights can only be done in a **Model** by using its `from_pretrained` method. But I agree that this could be improved in the documentation. I'll reopen this, try to improve the documentation, and close the issue when it's done.

Thanks for bringing this to the attention!

Hi, we tried to make it clear in the documentation by specifying [it in the `PretrainedConfig` class](https://huggingface.co/transformers/main_classes/configuration.html#pretrainedconfig). 

I think we could add this note to `AutoConfig` as well, as I doubt users using `AutoConfig` read the documentation of `PretrainedConfig` as well.

Oh, sorry @BramVanroy I didn't see you assigned it to yourself. Do you want to add the documentation note? Maybe you have additional ideas of where it should be added?

I think that another place to mention this note is under the [from-config](https://huggingface.co/transformers/model_doc/auto.html#transformers.AutoModel.from_config) method.

You're right, it would be nice to specify it there as well!

> Oh, sorry @BramVanroy I didn't see you assigned it to yourself. Do you want to add the documentation note? Maybe you have additional ideas of where it should be added?

Oh, go ahead! You know the library better than I do so your judgement of where to add a note is better.