Different Results from version 0.4.0 to version 0.5.0
Author: ShengleiH
Pull request: no

Hi, I found the results after training is different from version 0.4.0 to version 0.5.0. I have fixed all initialization to reproduce the results. And I also test version 0.2.0 and 0.3.0, the results are the same to version 0.4.0, but from version 0.5.0 +, the results is different. I am wondering that have you trained a new model, so the weights changed? 

Comments:
Hi, no we didn't change the weights. Can you share a sample on which the results are different?

Hi @thomwolf , thanks for your quick reply. I found even version 0.4.0 is different to version 0.2.0 and 0.3.0. I trained the model on v0.4.0, and then I tried to load the model using v0.2.0, here is the mismatch of keys:

```
RuntimeError: Error(s) in loading state_dict for BertForSequenceClassification:

Missing key(s) in state_dict: "bert.embeddings.LayerNorm.gamma", "bert.embeddings.LayerNorm.beta",...

Unexpected key(s) in state_dict: "bert.embeddings.LayerNorm.weight", "bert.embeddings.LayerNorm.bias",...
```
This is also appears when I trained the model on v0.5.0+ and tried to load the model using v0.2.0 and v0.3.0.

However, the first step loss are the same to v0.2.0, v0.3.0 and v0.4.0, loss=0.7228, but the first step loss are different to v0.5.0+ whose loss is 0.7091. And the final converge results are different too. I have set a seed to reproduce the results.

Oh, sorry, I found the mismatch problem is from my loading scripts (the 2nd method in [Serialization best-practices](https://github.com/huggingface/pytorch-pretrained-BERT#serialization-best-practices)), because I didn't use the mapping from old_keys to new_keys as you did in 'from_pretrained()' function.

Hi @thomwolf , I have found where the differences are. It is the different 'init_bert_weights' that makes the results different.

In version 0.2.0, 0.3.0 and 0.4.0, you use 'normal_' to initialize the 'BertLayerNorm':

```
def init_bert_weights(self, module):
        """ Initialize the weights.
        """
        if isinstance(module, (nn.Linear, nn.Embedding)):
            # Slightly different from the TF version which uses truncated_normal for initialization
            # cf https://github.com/pytorch/pytorch/pull/5617
            module.weight.data.normal_(mean=0.0, std=self.config.initializer_range)
        elif isinstance(module, BertLayerNorm):
            module.beta.data.normal_(mean=0.0, std=self.config.initializer_range)
            module.gamma.data.normal_(mean=0.0, std=self.config.initializer_range)
        if isinstance(module, nn.Linear) and module.bias is not None:
            module.bias.data.zero_()
```

but in version 0.5.0+, you use 'zeros_' and 'ones_' to initialize 'BertLayerNorm':

```
def init_bert_weights(self, module):
        """ Initialize the weights.
        """
        if isinstance(module, (nn.Linear, nn.Embedding)):
            # Slightly different from the TF version which uses truncated_normal for initialization
            # cf https://github.com/pytorch/pytorch/pull/5617
            module.weight.data.normal_(mean=0.0, std=self.config.initializer_range)
        elif isinstance(module, BertLayerNorm):
            module.bias.data.zero_()
            module.weight.data.fill_(1.0)
        if isinstance(module, nn.Linear) and module.bias is not None:
            module.bias.data.zero_()
```

By the way, after correct mapping from old_keys to new_keys, the old version pertained model could be loaded by new version with no different results! Thank you for sharing such great work with us!