convert_pytorch_checkpoint_to_tf2.py AttributeError: embeddings.word_embeddings.weight not found in PyTorch model
Author: ffaisal93
Pull request: no

I am trying to convert a finetuned bert model to tensorflow. The model was finetuned using pytorch-pretrained-bert on bert-base-multilingual-cased. But I am getting the following error while trying to convert using the tuned checkpoint.

code:
```
from transformers import convert_pytorch_checkpoint_to_tf2
convert_pytorch_checkpoint_to_tf2.convert_pt_checkpoint_to_tf("bert", "best_network.pt",
                                                             "bert-base-multilingual-cased",
                                                              "bert_aligned.ckpt")
```

error:
```
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-16-e0bc4c4758a1> in <module>
      1 convert_pytorch_checkpoint_to_tf2.convert_pt_checkpoint_to_tf("bert", "best_network.pt",
      2                                                              "bert-base-multilingual-cased",
----> 3                                                               "bert_aligned.ckpt")

~/opt/anaconda3/lib/python3.7/site-packages/transformers/convert_pytorch_checkpoint_to_tf2.py in convert_pt_checkpoint_to_tf(model_type, pytorch_checkpoint_path, config_file, tf_dump_path, compare_with_pt_model, use_cached_models)
    271         pytorch_checkpoint_path = cached_path(pytorch_checkpoint_url, force_download=not use_cached_models)
    272     # Load PyTorch checkpoint in tf2 model:
--> 273     tf_model = load_pytorch_checkpoint_in_tf2_model(tf_model, pytorch_checkpoint_path)
    274 
    275     if compare_with_pt_model:

~/opt/anaconda3/lib/python3.7/site-packages/transformers/modeling_tf_pytorch_utils.py in load_pytorch_checkpoint_in_tf2_model(tf_model, pytorch_checkpoint_path, tf_inputs, allow_missing_keys)
     91 
     92     return load_pytorch_weights_in_tf2_model(
---> 93         tf_model, pt_state_dict, tf_inputs=tf_inputs, allow_missing_keys=allow_missing_keys
     94     )
     95 

~/opt/anaconda3/lib/python3.7/site-packages/transformers/modeling_tf_pytorch_utils.py in load_pytorch_weights_in_tf2_model(tf_model, pt_state_dict, tf_inputs, allow_missing_keys)
    166                     continue
    167 
--> 168             raise AttributeError("{} not found in PyTorch model".format(name))
    169 
    170         array = pt_state_dict[name].numpy()

AttributeError: embeddings.word_embeddings.weight not found in PyTorch model


```

Comments:
Are you trying to convert it a HuggingFace TensorFlow object? If so can you do the following?

```
from transformers import TFBertForPreTraining

model = TFBertForPreTraining.from_pretrained(path_to_checkpoint, from_pt=True)
```

How did you fine-tune your model? How did you save it? Did you train it using HuggingFace transformers? Can you load it back in a PyTorch object or is it failing too?

@LysandreJik thanks for the reply.
this was pretrained to do multilingual alignment and trained using pytorch-pretrained-bert. Example code snippet:
```
def get_bert(bert_model, bert_do_lower_case):    
      from pytorch_pretrained_bert import BertTokenizer, BertModel    
      tokenizer = BertTokenizer.from_pretrained(bert_model, do_lower_case = bert_do_lower_case)    bert =                   
      BertModel.from_pretrained(bert_model)    return tokenizer, bert
class WordLevelBert(nn.Module):    
"""    Runs BERT on sentences but only keeps the last subword embedding for    each word.    """    
      def __init__(self, model, do_lower_case):        
           super().__init__()        
           self.bert_tokenizer, self.bert = get_bert(model, do_lower_case)        
           self.dim = self.bert.pooler.dense.in_features        
           self.max_len = self.bert.embeddings.position_embeddings.num_embeddings               
            if use_cuda:            
            self.cuda()        
    def forward(self, sentences, include_clssep = True):        
            batch_size = 128        
            ann_full = None        
            for i in range(0, len(sentences), batch_size):            
             ann = self.annotate(sentences[i:i+batch_size], include_clssep = include_clssep)
         .....
``` 

and I saved it in the following manner after training:
```
torch.save({'state_dict': model.state_dict(),  'trainer' : trainer.state_dict(),}, 'best_network.pt')
```

Update:
I could get rid of the error by making start_prefix_to_remove="" and by making pt_state_dict=pt_state_dict['state_dict'] in the file: https://github.com/huggingface/transformers/blob/master/src/transformers/modeling_tf_pytorch_utils.py

But now I get this new error:
```
~/opt/anaconda3/lib/python3.7/site-packages/transformers/modeling_tf_pytorch_utils.py in load_pytorch_checkpoint_in_tf2_model(tf_model, pytorch_checkpoint_path, tf_inputs, allow_missing_keys)
     92 
     93     return load_pytorch_weights_in_tf2_model(
---> 94         tf_model, pt_state_dict, tf_inputs=tf_inputs, allow_missing_keys=allow_missing_keys
     95     )
     96 

~/opt/anaconda3/lib/python3.7/site-packages/transformers/modeling_tf_pytorch_utils.py in load_pytorch_weights_in_tf2_model(tf_model, pt_state_dict, tf_inputs, allow_missing_keys)
    169                     continue
    170 
--> 171             raise AttributeError("{} not found in PyTorch model".format(name))
    172 
    173         array = pt_state_dict[name].numpy()

AttributeError: cls.seq_relationship.weight not found in PyTorch model

```

the fine tuned state dict can be loaded fine by:
```
from pytorch_pretrained_bert import BertTokenizer, BertModel
bert = BertModel.from_pretrained('bert-base-multilingual-cased')
tokenizer = BertTokenizer.from_pretrained('bert-base-multilingual-cased', do_lower_case = False)
bert.load_state_dict(torch.load('best_network.pt')['state_dict'])
```
And I got similar type of error while finetuning and saving a tutorial notebook this one https://www.kaggle.com/eggwhites2705/transformers-multi-label-classification

----------------------------------------------------------------------------------------------

I use this code now and got a tf model:
```
from transformers import TFBertModel
model = TFBertModel.from_pretrained("./demo_model", from_pt=True)
model.save("./demo_tf")
```

I got a .pb model and variable files like .data and .index but no .meta file. My aim is to use this .data and .index file insted of the original bert initial checkpoint in the tydiqa code: https://github.com/google-research-datasets/tydiqa/tree/master/baseline




Thank you for clarifying! In this case, if you have a PyTorch model that correctly loads (we recommend always using `from_pretrained`/`save_pretrained` rather than `.save()` and `torch.load`) that you want to convert to the "original" TensorFlow, then you should be able to use this script: 

https://github.com/huggingface/transformers/blob/master/src/transformers/models/bert/convert_bert_pytorch_checkpoint_to_original_tf.py

It will convert your model to a TF1 model usable in Google's original repository.

@LysandreJik thanks a lot. This solved my issue. Can you give me an idea about one thing: when I saved the tensorflow checkpoint with TFBertModel.from_pretrained("./demo_model", from_pt=True).save(..) the saved checkpoint was smaller in size like 411 mb. Today I got the right sized checkpoint with the script you suggested (711 mb)....why the size was smaller in previous approach?

I'm not entirely sure, but I guess it would make sense for the sizes to be different as the saving format is different between TFBertModel (TF2) and the TF1 saved checkpoint. Maybe our TF expert @Rocketknight1 has more insights :)

No idea, unfortunately! I don't **think** the format changed that massively between TF1 and TF2.

One things that strikes me is that 711mb for a bert-base model is quite large: With 110M parameters, we should expect it to take up about 110*4 = 440mb of space uncompressed, because each parameter is a 32-bit (4-byte) float. That said, if it works, don't question it™

THANKS @LysandreJik AND @Rocketknight1 . This is helpful. Yes it took 440 earlier but when I used the tf1 conversion script it took 711 mb. I will finetune tydiqa on both of these models and let's see, if there is any difference any performence.