Cant import my pretrained bert model from NVIDIA/DeepLearningExamples/
Author: Limtle
Pull request: no
Labels: name: wontfix

name: Migration


 Trained with **NVIDIA/DeepLearningExamples/** and got my_bert_model/ckpt_1000.pt

**.pt to .bin**

 Edit my_bert_model/ckpt_1000.pt  to my_bert_model/pytorch_model.bin

**My code**
```configuration = BertConfig(3258, 768, 12, 12, 3072)
model = BertModel(configuration)
model = model.from_pretrained('/home/my_bert_model', state_dict)
for key, weight in model.state_dict().items():
    print (weight)
```
**Output**
Every execution of this code produces different ouput


Comments:
Try:

```
model.eval()
```

before you access the weights.

Its seems not work. 
```
embeddings.word_embeddings.weight
tensor([[ 0.0073,  0.0080,  0.0307,  ..., -0.0172,  0.0148, -0.0401],
        [-0.0271,  0.0110,  0.0011,  ..., -0.0079,  0.0236, -0.0037],
        [ 0.0005,  0.0066, -0.0009,  ..., -0.0065,  0.0167,  0.0301],
        ...,
        [ 0.0062, -0.0385, -0.0091,  ..., -0.0022,  0.0043,  0.0018],
        [-0.0188,  0.0154, -0.0023,  ..., -0.0049, -0.0108,  0.0393],
        [-0.0257, -0.0056,  0.0155,  ..., -0.0198,  0.0280, -0.0143]])```
```
```
embeddings.word_embeddings.weight
tensor([[-0.0011, -0.0345,  0.0094,  ...,  0.0024,  0.0229,  0.0194],
        [-0.0246,  0.0329,  0.0231,  ...,  0.0436,  0.0246, -0.0012],
        [ 0.0069,  0.0050, -0.0020,  ..., -0.0002,  0.0043,  0.0208],
        ...,
        [-0.0139, -0.0091,  0.0110,  ..., -0.0128, -0.0015, -0.0027],
        [ 0.0297,  0.0063, -0.0066,  ...,  0.0070,  0.0157,  0.0417],
        [-0.0341,  0.0458,  0.0054,  ..., -0.0525,  0.0003, -0.0122]])
```

Solved! I find the layer_name of NVIDIA/DeepLearningExamples/  is dismatch  huggingface/transformers

@Limtle  This seems like it is important information. What exactly do you mean? The layer names are not the same in NVIDIA's examples and the layers here in the HuggingFace repo?

I print the weights_name of bertmodel trained with NVIDIA/DeepLearningExamples/
```
bert.embeddings.word_embeddings.weight
bert.embeddings.position_embeddings.weight
bert.embeddings.token_type_embeddings.weight
...
bert.encoder.layer.11.output.dense.weight
bert.encoder.layer.11.output.dense.bias
bert.encoder.layer.11.output.LayerNorm.weight
bert.encoder.layer.11.output.LayerNorm.bias
bert.pooler.dense_act.weight
bert.pooler.dense_act.bias
cls.predictions.bias
cls.predictions.transform.dense_act.weight
cls.predictions.transform.dense_act.bias
cls.predictions.transform.LayerNorm.weight
cls.predictions.transform.LayerNorm.bias
cls.predictions.decoder.weight
cls.seq_relationship.weight
cls.seq_relationship.bias
```
And the weights_name of huggingface/transformers/BertModel()
```
embeddings.word_embeddings.weight
embeddings.position_embeddings.weight
embeddings.token_type_embeddings.weight
...
encoder.layer.11.output.dense.weight
encoder.layer.11.output.dense.bias
encoder.layer.11.output.LayerNorm.weight
encoder.layer.11.output.LayerNorm.bias
pooler.dense.weight
pooler.dense.bias
```
This is my code 
load model from NVIDIA/DeepLearningExamples/ to huggingface/transformers
```
configuration = BertConfig.from_json_file(config_path)
tokenizer = BertTokenizer.from_pretrained(vocab_path)
model = BertModel(configuration)
state_dict = {k.replace('bert.','').replace('.dense_act','.dense'):v for k,v in torch.load(os.path.join(pytorch_pretrained_model_path, 'pytorch_model.bin'))['model'].items()}
model.load_state_dict(state_dict, strict= False)
#model = model.from_pretrained(pytorch_pretrained_model_path, state_dict= state_dict)
```

> Is it the right way to solve this problem?



This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.