Loading fine_tuned BertModel fails due to prefix error
Author: sebastianruder
Pull request: no

I am loading a pretrained BERT model with `BertModel.from_pretrained` as I feed the `pooled_output` representation directly to a loss without a head. After fine-tuning the model, I save it as in [`run_classifier.py`](https://github.com/huggingface/pytorch-pretrained-BERT/blob/master/examples/run_classifier.py#L553). 
Afterwards, I want to load the fine-tuned model, again without a head, so I'm using `BertModel.from_pretrained` model again to initialize it, this time from the directory where the config and model files are stored. 
When trying to load the pretrained model, none of the weights are found and I get:
```
Weights of BertModel not initialized from pretrained model: ['bert.embeddings.word_embeddings.weight'
, 'bert.embeddings.position_embeddings.weight', 'bert.embeddings.token_type_embeddings.weight', 'bert
.embeddings.LayerNorm.weight', 'bert.embeddings.LayerNorm.bias', 'bert.encoder.layer.0.attention.self
.query.weight', 'bert.encoder.layer.0.attention.self.query.bias', 'bert.encoder.layer.0.attention.self.key.weight', ...]
```
This seems to be due to [this line](https://github.com/huggingface/pytorch-pretrained-BERT/blob/master/pytorch_pretrained_bert/modeling.py#L543) in `modeling.py`. As `BertModel.from_pretrained` does not create a `bert` attribute (in contrast to the BertModels with a head), the `bert.` prefix is used erroneously instead of the `''` prefix, which causes the weights of the fine-tuned model not to be found.
If I change this line to check additionally if we load a fine-tuned model, then this works:
```
load(model, prefix='' if hasattr(model, 'bert') or pretrained_model_name not in PRETRAINED_MODEL_ARCHIVE_MAP else 'bert.')
```
Does this make sense? Let me know if I'm using `BertModel.from_pretrained` in the wrong way or if I should be using a different model for fine-tuning if I just care about the `pooled_output` representation.

Comments:
I think that you have find the problem but I'm not sure if your fix is the most appropriate way to deal with it. As this problem will only happen when we are loading a `BertModel` pretrained instance, maybe 

```
load(model, prefix='' if hasattr(model, 'bert') or cls == BertModel else 'bert.')
```
would be more logical. Could you check if this change also fixes your problem?

The problem is that this only happens when we load a `BertModel` that was previously fine-tuned. If we load a pretrained `BertModel`, then the pretrained parameters don't have the `bert.` prefix, so we have to add it and it works. However, if we load the fine-tuned `BertModel`, then the parameters already have the `bert.` prefix, so we don't need to add it anymore. But this is not recognized at the moment.
So the above change causes the loading of a pretrained `BertModel` to fail.

I tried to reproduce your problem to better understand but I'm getting some pretty strange results. I don't get any error but the weights do not load properly. Am I missing something obvious or is it more or less what you are doing?

```python
import torch
from pytorch_pretrained_bert import BertModel
from pytorch_pretrained_bert.file_utils import PYTORCH_PRETRAINED_BERT_CACHE

bert_model = 'bert-base-uncased'
save_file = 'test_ruder/model.bin'
model_base = BertModel.from_pretrained(
    bert_model,
    cache_dir=PYTORCH_PRETRAINED_BERT_CACHE / 'distributed_{}'.format(-1)
)

# Saving
model_to_save = model_base.module if hasattr(model_base, 'module') else model_base
torch.save(model_to_save.state_dict(), save_file)

# Loading
model_state_dict = torch.load(save_file)
model_loaded = BertModel.from_pretrained(
    bert_model,
    state_dict = model_state_dict
)

# Tests
param_orig = list(model_base.parameters())
param_load = list(model_loaded.parameters())
print(len(param_orig) == len(param_load)) # True
print(all(x.shape == y.shape for x, y in zip(param_orig, param_load))) # True
for p_orig, p_load in zip(param_orig, param_load):
    print(torch.all(p_orig == p_load)) # prints tensor(0, dtype=torch.uint8) everytime
```

Thanks for adding this working example. Yep, that's the issue I'm facing. I've slightly amended it to load the model from the config and weights file in the archive instead:
```python
import torch
from pytorch_pretrained_bert import BertModel, modeling
from pytorch_pretrained_bert.file_utils import PYTORCH_PRETRAINED_BERT_CACHE
from pathlib import Path

save_dir = Path('test_ruder')
save_dir.mkdir(exist_ok=True)
bert_model = 'bert-base-uncased'
save_file = save_dir / modeling.WEIGHTS_NAME
config_file = save_dir / modeling.CONFIG_NAME
model_base = BertModel.from_pretrained(
    bert_model,
    cache_dir=PYTORCH_PRETRAINED_BERT_CACHE / 'distributed_{}'.format(-1)
)

# Saving
model_to_save = model_base.module if hasattr(model_base, 'module') else model_base
torch.save(model_to_save.state_dict(), save_file)
with open(config_file, 'w') as f:
   f.write(model_base.config.to_json_string())

# Loading
model_state_dict = torch.load(save_file)
model_loaded = BertModel.from_pretrained(save_dir)

# Tests
param_orig = list(model_base.parameters())
param_load = list(model_loaded.parameters())
print(len(param_orig) == len(param_load)) # True
print(all(x.shape == y.shape for x, y in zip(param_orig, param_load))) # True
for p_orig, p_load in zip(param_orig, param_load):
    print(torch.all(p_orig == p_load)) # prints tensor(0, dtype=torch.uint8) everytime
```

I don't get the warnings that you are mentioning in your original post, the piece of code that I've created seems to fail for another reason. Could you please try to reproduce your original problem in a minimal piece of code?

That's the message printed by the logger [here](https://github.com/huggingface/pytorch-pretrained-BERT/blob/master/pytorch_pretrained_bert/modeling.py#L545). You need to enable logging first. You can also just print the same message instead.

Hi sebastian, indeed, the pretrained loading script is currently designed to load the weights from `BertForPreTraining ` models.

I will fix that in the next release. We just have to slightly modify [the line you indicated](https://github.com/huggingface/pytorch-pretrained-BERT/blob/master/pytorch_pretrained_bert/modeling.py#L543) to check the keys of the state dictionary:
```python
load(model, prefix='bert.' if not hasattr(model, 'bert') and any(s.startwith('bert.') in state_dict.keys()) else '')
```

In the meantime you can fix your problem by adding `bert.` to the keys of your state dictionary.
In your example, you can either change the saving operation:
```python
# Saving
model_to_save = model_base.module if hasattr(model_base, 'module') else model_base
to_save_dict = model_to_save.state_dict()
to_save_with_prefix = {}
for key, value in to_save_dict.items():
    to_save_with_prefix['bert.' + key] = value
torch.save(to_save_with_prefix, save_file)
with open(config_file, 'w') as f:
   f.write(model_base.config.to_json_string())
```
or the loading operation:
```python
# Loading
model_state_dict = torch.load(save_file)
state_dict_with_prefix = {}
for key, value in model_state_dict.items():
    state_dict_with_prefix['bert.' + key] = value
model_loaded = BertModel.from_pretrained(save_dir, state_dict=state_dict_with_prefix)
```

Actually Sebastian, since the model you save and the model you load are instances of the same `BertModel` class, you can also simply use the standard PyTorch serialization practice (we only have a special `from_pretrained` loading function to be able to load various type of models using the same pre-trained model stored on AWS).

Just build a new `BertModel` using the configuration file you saved.

Here is a snippet :
```python
# Saving (same as you did)
model_to_save = model_base.module if hasattr(model_base, 'module') else model_base
torch.save(model_to_save.state_dict(), save_file)
with open(config_file, 'w') as f:
   f.write(model_base.config.to_json_string())

# Loading (using standard PyTorch loading practice)
config = BertConfig(config_file)
model = BertModel(config)
model.load_state_dict(torch.load(save_file))
```

Thanks a lot for the comprehensive suggestions, @thomwolf. You're totally right that just loading it as normally in PyTorch is the most straightforward and simplest way. Your last suggestion works. Thanks! 👍 

Hi All,

iam facing following issue while loading pretrained BERT Sequence model with my own data

RuntimeError: Error(s) in loading state_dict for DataParallel:
	Missing key(s) in state_dict: "module.out.weight", "module.out.bias". 
	Unexpected key(s) in state_dict: "bert.embeddings.word_embeddings.weight", "bert.embeddings.position_embeddings.weight", "bert.embeddings.token_type_embeddings.weight", "bert.embeddings.LayerNorm.weight", "bert.embeddings.LayerNorm.bias", "bert.encoder.layer.0.attention.self.query.weight", "bert.encoder.layer.0.attention.self.query.bias", "bert.encoder.layer.0.attention.self.key.weight", "bert.encoder.layer.0.attention.self.key.bias", "bert.encoder.layer.0.attention.self.value.weight", "bert.encoder.layer.0.attention.self.value.bias", "bert.encoder.layer.0.attention.output.dense.weight", "bert.encoder.layer.0.attention.output.dense.bias", "bert.encoder.layer.0.attention.output.LayerNorm.weight", "bert.encoder.layer.0.attention.output.LayerNorm.bias", "bert.encoder.layer.0.intermediate.dense.weight", "bert.encoder.layer.0.intermediate.dense.bias", "bert.encoder.layer.0.output.dense.weight", "bert.encoder.layer.0.output.dense.bias", "bert.encoder.layer.0.output.LayerNorm.weight", "bert.encoder.layer.0.output.LayerNorm.bias", "bert.encoder.layer.1.attention.self.query.weight", "bert.encoder.layer.1.attention.self.query.bias", "bert.encoder.layer.1.attention.self.key.weight", "bert.encoder.layer.1.attention.self.key.bias", "bert.encoder.layer.1.attention.self.value.weight", "bert.encoder.layer.1.attention.self.value.bias", "bert.encoder.layer.1.attention.output.dense.weight", "bert.encoder.layer.1.attention.output.dense.bias", "bert.encoder.layer.1.attention.output.LayerNorm....


any idea about this error