convert_gpt2_checkpoint_to_pytorch dimensions assertion error
Author: chrisyue1998
Pull request: no

I finetuned a GPT-2 model using TensorFlow (using https://github.com/nshepperd/gpt-2), and I tried to run the TF to PyTorch conversion script, but I got this error:

`Traceback (most recent call last):
  File "convert_gpt2_checkpoint_to_pytorch.py", line 72, in <module>
    args.pytorch_dump_folder_path)
  File "convert_gpt2_checkpoint_to_pytorch.py", line 39, in convert_gpt2_checkpoint_to_pytorch
    load_tf_weights_in_gpt2(model, gpt2_checkpoint_path)
  File "/Users/UAC897/Documents/kepler/venv/lib/python3.7/site-packages/pytorch_pretrained_bert/modeling_gpt2.py", line 90, in load_tf_weights_in_gpt2
    assert pointer.shape == array.shape
AssertionError: (torch.Size([2304]), (3072,))`

Does anyone have any ideas?

Comments:
Hey, I tried doing the same and was successful(in running the script, at least).
What do you specify as your --gpt2_checkpoint_path as? (hope you have copied your finetuned checkpoints to preloaded 117M)?

Update: Currently the output just stores a `config.json` and  `pytorch_model.bin`. Suprisingly I don't see the vocab.txt and the merges.txt that they specify in the README. My tensorflow model checkpoint has a vocab.bpe but not merges.txt. So I'm stuck on how to proceed with the following error when running `run_gpt2.py`:
` We assumed '/home/code-base/pytorch-pretrained-BERT/pytorch_pretrained_bert/temp-model/' was a path or url but couldn't find files /home/code-base/pytorch-pretrained-BERT/pytorch_pretrained_bert/temp-model/vocab.json and /home/code-base/pytorch-pretrained-BERT/pytorch_pretrained_bert/temp-model/merges.txt at this path or url.`
Since I'm unable to generate samples using these weights now, any ideas would be great!

I was able to fix my issue. I realized that the GPTConfig constructor used in convert_gpt2_checkpoint_to_pytorch.py is only for the 117M model, while I was trying to convert a 345M model. I ended up just making a new json file for the larger model. I never ran into your issues, however, as I didn't use run_gpt2.py.

> I was able to fix my issue. I realized that the GPTConfig constructor used in convert_gpt2_checkpoint_to_pytorch.py is only for the 117M model, while I was trying to convert a 345M model. I ended up just making a new json file for the larger model. I never ran into your issues, however, as I didn't use run_gpt2.py.

So the new checkpoint files for you contain only a `config.json` and `pytorch_model.bin`?

> > I was able to fix my issue. I realized that the GPTConfig constructor used in convert_gpt2_checkpoint_to_pytorch.py is only for the 117M model, while I was trying to convert a 345M model. I ended up just making a new json file for the larger model. I never ran into your issues, however, as I didn't use run_gpt2.py.
> 
> So the new checkpoint files for you contain only a `config.json` and `pytorch_model.bin`?

Correct

How did you end up fixing this problem? What json file did you have to create? I'd also like to convert a 345M model and am running into the problem described in this issue.

@dsonbill Sorry, I honestly don't remember. My work was on the laptop I used at my old job, so I don't have access to the files anymore. You could check my really old fork though and maybe find something useful there.

Hacky solution by modifying this script: https://raw.githubusercontent.com/huggingface/transformers/master/src/transformers/convert_gpt2_original_tf_checkpoint_to_pytorch.py

Before:
```py
    # Construct model
    if gpt2_config_file == "":
        config = GPT2Config()
    else:
        config = GPT2Config.from_json_file(gpt2_config_file)
    model = GPT2Model(config)
```

After:

```py
    # Construct model
    config = GPT2Config.from_pretrained('gpt2-medium') # Replace 'gpt2-medium' with whichever model spec you're converting
    model = GPT2Model(config)
```

> Hacky solution by modifying this script: https://raw.githubusercontent.com/huggingface/transformers/master/src/transformers/convert_gpt2_original_tf_checkpoint_to_pytorch.py
> 
> Before:
> 
> ```python
>     # Construct model
>     if gpt2_config_file == "":
>         config = GPT2Config()
>     else:
>         config = GPT2Config.from_json_file(gpt2_config_file)
>     model = GPT2Model(config)
> ```
> 
> After:
> 
> ```python
>     # Construct model
>     config = GPT2Config.from_pretrained('gpt2-medium') # Replace 'gpt2-medium' with whichever model spec you're converting
>     model = GPT2Model(config)
> ```

Hi, this doesn't work on me. I'm facing the same problem. Does anyone have any ideas?

> > Hacky solution by modifying this script: https://raw.githubusercontent.com/huggingface/transformers/master/src/transformers/convert_gpt2_original_tf_checkpoint_to_pytorch.py
> > Before:
> > ```python
> >     # Construct model
> >     if gpt2_config_file == "":
> >         config = GPT2Config()
> >     else:
> >         config = GPT2Config.from_json_file(gpt2_config_file)
> >     model = GPT2Model(config)
> > ```
> > 
> > 
> > After:
> > ```python
> >     # Construct model
> >     config = GPT2Config.from_pretrained('gpt2-medium') # Replace 'gpt2-medium' with whichever model spec you're converting
> >     model = GPT2Model(config)
> > ```
> 
> Hi, this doesn't work on me. I'm facing the same problem. Does anyone have any ideas?

I could get past the initial error by setting the config to be hparams.json

```export OPENAI_GPT2_CHECKPOINT_PATH=gpt2/355M

transformers-cli convert --model_type gpt2 \
  --tf_checkpoint $OPENAI_GPT2_CHECKPOINT_PATH/model.ckpt \
  --pytorch_dump_output $OPENAI_GPT2_CHECKPOINT_PATH \
  --config $OPENAI_GPT2_CHECKPOINT_PATH/hparams.json
```

which seems to create the model checkpoint.   I haven't check to see if the converted model works yet though.