Bert model instantiated from BertForMaskedLM.from_pretrained('bert-base-uncased') and BertForMaskedLM(BertConfig.from_pretrained('bert-base-uncased')) give different results 
Author: christian-storm
Pull request: no
Labels: name: wontfix

The two different methods for instantiating a model produce different losses.

`from pytorch_transformers import BertForMaskedLM, BertConfig, BertTokenizer
import torch

tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
input_ids = torch.tensor(tokenizer.encode("Hello, my dog is cute")).unsqueeze(0)  

config = BertConfig.from_pretrained('bert-base-uncased')
config_model = BertForMaskedLM(config)
config_model.eval()
with torch.no_grad():
    config_outputs = config_model(input_ids, masked_lm_labels=input_ids)
config_loss = config_outputs[0]
print(config_loss.item())

pretrained_model = BertForMaskedLM.from_pretrained('bert-base-uncased')
pretrained_model.eval()
with torch.no_grad():
    pretrained_outputs = pretrained_model(input_ids, masked_lm_labels=input_ids)
pretrained_loss = pretrained_outputs[0]
print(pretrained_loss.item())
assert config_loss.item() == pretrained_loss.item()`

The losses produced:
10.574708938598633
1.690806269645691


Comments:
Hi, not only they give different results, but also BertModel(BertConfig.from_pretrained('bert-base-uncased')) will give a different result each time you run it. **Other bert models also have this problem**; I think this is a bug. @thomwolf 

Following code works well and produce the same result each time you run it.
___________________________
import torch
from pytorch_transformers import BertTokenizer, BertModel, BertConfig
import numpy as np

tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
model = BertModel.from_pretrained('bert-base-uncased')
model.eval()
input_ids = torch.tensor(tokenizer.encode("Hello, my dog is cute")).unsqueeze(0)  # Batch size 1
outputs = model(input_ids)
a = np.squeeze(outputs[0].detach().numpy())
avg = np.mean(a,axis = 0)
print(avg[0])


**Above code will always output -0.2769656.**
___________________________
Following code will produce a different result each time:
__________________________
config = BertConfig.from_pretrained('bert-base-uncased')
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
model = BertModel(config)
model.eval()
input_ids = torch.tensor(tokenizer.encode("Hello, my dog is cute")).unsqueeze(0) 
outputs = model(input_ids)
last_hidden_states = outputs[0]
a = np.squeeze(last_hidden_states[0].detach().numpy())
avg = np.mean(a,axis = 0)
print(avg[0])

It seems that BertModel(config) returns a random intialized Bert model with the architecture as the config file indicates, because the __init__ function doesn't load pretrained weights. BertModel.from_pretrained() is the right function to load both model architecture and pretrained weights. It's the same for other bert classes.

Shouldn't BertConfig.from_pretrained('bert-base-uncased') return a config that loads pretrained weights instead of randomly initialized ones?  I thought was the whole point of the example code in the [docs](https://huggingface.co/pytorch-transformers/model_doc/bert.html#bertformaskedlm):


config = BertConfig.from_pretrained('bert-base-uncased')
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')

model = BertForMaskedLM(config)
input_ids = torch.tensor(tokenizer.encode("Hello, my dog is cute")).unsqueeze(0)  # Batch size 1
outputs = model(input_ids, masked_lm_labels=input_ids)
loss, prediction_scores = outputs[:2]

It should return a config that loads pretrained models; however it does not act like this way.


Sent from Yahoo Mail for iPhone


On Sunday, August 4, 2019, 4:32 PM, Christian Storm <notifications@github.com> wrote:


Shouldn't BertConfig.from_pretrained('bert-base-uncased') return a config that loads pretrained weights instead of randomly initialized ones? I thought was the whole point of the example code in the docs:

config = BertConfig.from_pretrained('bert-base-uncased')
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')

model = BertForMaskedLM(config)
input_ids = torch.tensor(tokenizer.encode("Hello, my dog is cute")).unsqueeze(0) # Batch size 1
outputs = model(input_ids, masked_lm_labels=input_ids)
loss, prediction_scores = outputs[:2]

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or mute the thread.





You are right, the example in the doc is misleading.

The only way to load pretrained weights in a `model` is to call a `model_class.from_pretrained()` method. I'll fix the doc.

I've fixed the examples of loading pretrained models in docstrings :-) #973 

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.