'BertEmbeddings' object has no attribute 'bias' while converting tf checkpoint
Author: blueberry-cake
Pull request: no

When trying to convert the checkpoint of a self pre-trained tensorflow BERT model (using the **[create-pretraining.py][1]** script from google) into a pytorch model using **convert_bert_original_tf_checkpoint_to_pytorch.py**

I always end up with the following error:
**AttributeError: 'BertEmbeddings' object has no attribute 'bias'**

The init_vars names (just the first ones) look like this:

```
['bert/embeddings/layer_normalization/beta', 'bert/embeddings/layer_normalization/beta/adam_m', 'bert/embeddings/layer_normalization/beta/adam_v', 'bert/embeddings/layer_normalization/gamma', 'bert/embeddings/layer_normalization/gamma/adam_m', 'bert/embeddings/layer_normalization/gamma/adam_v']
```
Code that produces the error looks like this:
```
for m_name in name:                                                     
            if re.fullmatch(r"[A-Za-z]+_\d+", m_name):                          
                scope_names = re.split(r"_(\d+)", m_name)                       
            else:                                                               
                scope_names = [m_name]                                          
            if scope_names[0] == "kernel" or scope_names[0] == "gamma":         
                pointer = getattr(pointer, "weight")                            
            elif scope_names[0] == "output_bias" or scope_names[0] == "beta":   
                print(scope_names)                                              
                pointer = getattr(pointer, "bias")                              
            elif scope_names[0] == "output_weights":                            
                pointer = getattr(pointer, "weight")                            
            elif scope_names[0] == "squad":                                     
                pointer = getattr(pointer, "classifier")                        
            else:                                                               
                try:                                                            
                    pointer = getattr(pointer, scope_names[0])                  
                except AttributeError:                                          
                    logger.info("Skipping {}".format("/".join(name)))
```
Going through all the names and getting the right attributes from the model. When it comes to the Layer Normalization in the BertEmbeddings the script produces an error. Did anyone else encouter that error before? How did you fix this? Did I mix something up with the tensorflow versions? Thanks for your help in advance!

Here again the whole stacktrace:
```
Traceback (most recent call last):
  File "convert_bert_original_tf_checkpoint_to_pytorch.py", line 62, in <module>
    convert_tf_checkpoint_to_pytorch(args.tf_checkpoint_path, args.bert_config_file, args.pytorch_dump_path)
  File "convert_bert_original_tf_checkpoint_to_pytorch.py", line 37, in convert_tf_checkpoint_to_pytorch
    load_tf_weights_in_bert(model, config, tf_checkpoint_path)
  File "/modeling_bert.py", line 136, in load_tf_weights_in_bert
    pointer = getattr(pointer, "bias")
  File "module.py", line 594, in __getattr__
    type(self).__name__, name))
AttributeError: 'BertEmbeddings' object has no attribute 'bias'
```
Bert Config is the following:

```
Building PyTorch model from configuration: BertConfig {
  "attention_probs_dropout_prob": 0.1,
  "gradient_checkpointing": false,
  "hidden_act": "gelu",
  "hidden_dropout_prob": 0.1,
  "hidden_size": 512,
  "initializer_range": 0.02,
  "intermediate_size": 2048,
  "layer_norm_eps": 1e-12,
  "max_position_embeddings": 512,
  "model_type": "bert",
  "num_attention_heads": 8,
  "num_hidden_layers": 8,
  "pad_token_id": 0,
  "type_vocab_size": 2,
  "vocab_size": 30522
}
```

  [1]: https://github.com/google-research/bert/blob/master/run_pretraining.py

Comments:
It was just the naming of "layer_norm" instead of "LayerNorm" I changed the script and now it works.

@blueberry-cake which script was that naming in? 

@blueberry-cake could you tell me the details of how you solve this problem? I have this problem, too，I do not understand the word "It was just the naming of "layer_norm" instead of "LayerNorm" I changed the script and now it works." Thanks for your help in advance！

Hi, I encountered the same problem. I spent quite a while googling online but didn't get a solution. Could you please let me know if you get the solution? @blueberry-cake @roxannemiller @ankunw 

> Hi, I encountered the same problem. I spent quite a while googling online but didn't get a solution. Could you please let me know if you get the solution? @blueberry-cake @roxannemiller @ankunw

maybe you could use he latest transformer have a try

No it still doesn't work. Sign :(

So I solved this problem with other people's help. Basically, I need to change the key name in my tf1 checkpoints. Here is the code. For further details, please see: https://colab.research.google.com/github/tensorflow/docs/blob/master/site/en/guide/migrate/migrating_checkpoints.ipynb?hl=id#scrollTo=NPQsXQveuQiC

```
import re
def change_name(checkpoint_path, output_prefix):
  ckpt = tf.train.Checkpoint(vars={name: variable})  
  ckpt.restore(converted_ckpt_path)
  """
  Args:
    checkpoint_path: Path to the TF1 checkpoint.
    output_prefix: Path prefix to the converted checkpoint.

  Returns:
    Path to the converted checkpoint.
  """
  vars = {}
  reader = tf.train.load_checkpoint(checkpoint_path)
  dtypes = reader.get_variable_to_dtype_map()

  for key in dtypes.keys():
    new_key = key
    if key=='bert/embeddings/layer_normalization/beta' or key=='bert/embeddings/layer_normalization/gamma':
      new_key=key.replace('layer_normalization','LayerNorm')
    elif re.search('layer_normalization_+\d+',key):
      new_key = re.sub('layer_normalization_+\d+','LayerNorm',key)
    elif re.search('layer_normalization',key):
      new_key = re.sub('layer_normalization','LayerNorm',key)
    print(new_key)
    vars[new_key] = tf.Variable(reader.get_tensor(key))
    
  return tf1.train.Saver(var_list=vars).save(sess=None, save_path=output_prefix)

> So I solved this problem with other people's help. Basically, I need to change the key name in my tf1 checkpoints. Here is the code. For further details, please see: https://colab.research.google.com/github/tensorflow/docs/blob/master/site/en/guide/migrate/migrating_checkpoints.ipynb?hl=id#scrollTo=NPQsXQveuQiC
> 
> ```
> import re
> def change_name(checkpoint_path, output_prefix):
>   ckpt = tf.train.Checkpoint(vars={name: variable})  
>   ckpt.restore(converted_ckpt_path)
>   """
>   Args:
>     checkpoint_path: Path to the TF1 checkpoint.
>     output_prefix: Path prefix to the converted checkpoint.
> 
>   Returns:
>     Path to the converted checkpoint.
>   """
>   vars = {}
>   reader = tf.train.load_checkpoint(checkpoint_path)
>   dtypes = reader.get_variable_to_dtype_map()
> 
>   for key in dtypes.keys():
>     new_key = key
>     if key=='bert/embeddings/layer_normalization/beta' or key=='bert/embeddings/layer_normalization/gamma':
>       new_key=key.replace('layer_normalization','LayerNorm')
>     elif re.search('layer_normalization_+\d+',key):
>       new_key = re.sub('layer_normalization_+\d+','LayerNorm',key)
>     elif re.search('layer_normalization',key):
>       new_key = re.sub('layer_normalization','LayerNorm',key)
>     print(new_key)
>     vars[new_key] = tf.Variable(reader.get_tensor(key))
>     
>   return tf1.train.Saver(var_list=vars).save(sess=None, save_path=output_prefix)
> ```

Dear friend, is there a complete integration of your code in "convert_bert_original_tf_checkpoint_to_pytorch.py"? I don't know how to adjust it using your code.