Converting tf weights:  AttributeError: 'GPT2Model' object has no attribute 'zeLoss'
Author: fabrahman
Pull request: no
Labels: name: wontfix

name: Migration

# 🐛 Bug

## Information

Model I am using: gpt-2
I am trying to convert a fine-tuned gpt-2 model on tensroflow to pytorch state_dict. I used the nice script [here](https://github.com/huggingface/transformers/blob/ce50305e5b8c8748b81b0c8f5539a337b6a995b9/src/transformers/convert_gpt2_original_tf_checkpoint_to_pytorch.py).


## To reproduce

Steps to reproduce the behavior:

1. I run the following command:
```
python convert_gpt2_original_tf_checkpoint_to_pytorch.py --gpt2_checkpoint_path /home/finetune/model_best.ckpt --pytorch_dump_folder_path /home/gpt2-fine-tf2torch/
```
after running this command, I see the logs showing that the tf weights are being loaded but suddently I got the following error:

```
INFO:transformers.modeling_gpt2:Loading TF weight transformer_decoder/layer_9/self_attention/multihead_attention/value/bias with shape [1024]
INFO:transformers.modeling_gpt2:Loading TF weight transformer_decoder/layer_9/self_attention/multihead_attention/value/kernel with shape [1024, 1024]
INFO:transformers.modeling_gpt2:Loading TF weight word_embedder/w with shape [50257, 1024]
INFO:transformers.modeling_gpt2:Loading TF weight word_embedder_1/w with shape [50257, 1024]
Traceback (most recent call last):
  File "convert_gpt2_original_tf_checkpoint_to_pytorch.py", line 67, in <module>
    convert_gpt2_checkpoint_to_pytorch(args.gpt2_checkpoint_path, args.gpt2_config_file, args.pytorch_dump_folder_path)
  File "convert_gpt2_original_tf_checkpoint_to_pytorch.py", line 38, in convert_gpt2_checkpoint_to_pytorch
    load_tf_weights_in_gpt2(model, config, gpt2_checkpoint_path)
  File "/home/anaconda3/envs/torch03/lib/python3.6/site-packages/transformers/modeling_gpt2.py", line 84, in load_tf_weights_in_gpt2
    pointer = getattr(pointer, scope_names[0])
  File "/home/anaconda3/envs/torch03/lib/python3.6/site-packages/torch/nn/modules/module.py", line 585, in __getattr__
    type(self).__name__, name))
AttributeError: 'GPT2Model' object has no attribute 'zeLoss'
```

I tried to load the tf checkpoint and inspect variables, there were no 'zeLoss' but ''OptimizeLoss' which I guess the script mistakenly matched only the 'zeLoss':

```
>>> import tensorflow as tf
>>> import os
>>> tf_path = os.path.abspath('model_best.ckpt')
>>> tf_vars = tf.train.list_variables(tf_path)
```
here is part of the ```tf_vars```:
```
[('OptimizeLoss/beta1_power', []), ('OptimizeLoss/beta2_power', []), ('OptimizeLoss/position_embedder/w/Adam', [1024, 1024]), ('OptimizeLoss/position_embedder/w/Adam_1', [1024, 1024]), ('OptimizeLoss/transformer_decoder/beta/Adam', [1024]), ('OptimizeLoss/transformer_decoder/beta/Adam_1', [1024]), ('OptimizeLoss/transformer_decoder/gamma/Adam', [1024]), ('OptimizeLoss/transformer_decoder/gamma/Adam_1', [1024]), ('OptimizeLoss/transformer_decoder/layer_0/beta/Adam', [1024]), ('OptimizeLoss/transformer_decoder/layer_0/beta/Adam_1', [1024]), ('OptimizeLoss/transformer_decoder/layer_0/ffn/conv1/bias/Adam', [4096]), ('OptimizeLoss/transformer_decoder/layer_0/ffn/conv1/bias/Adam_1', [4096]), ('OptimizeLoss/transformer_decoder/layer_0/ffn/conv1/kernel/Adam', [1024, 4096]),
```

I would appreciate if you can help me fix this .
Thanks


Comments:
@LysandreJik Any update on this? Thanks

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.