size mismatch for lm_head.decoder.weight
Author: Wingie
Pull request: no
Labels: name: wontfix

Hi i'm new to this,

first i started a finetune job
```
export ROC_STORIES_DIR=roc/

python run_openai_gpt.py \
  --model_name openai-gpt \
  --do_train \
  --do_eval \
  --train_dataset $ROC_STORIES_DIR/cloze_test_val__spring2016\ -\ cloze_test_ALL_val.csv \
  --eval_dataset $ROC_STORIES_DIR/cloze_test_test__spring2016\ -\ cloze_test_ALL_test.csv \
  --output_dir ../roc_gpt \
  --train_batch_size 16 \
```

i have the following files in the output folder
```
config.json  eval_results.txt  merges.txt  pytorch_model.bin  special_tokens.txt  vocab.json
```

however when i run 
```
python run_gpt2.py --model_name_or_path=../roc_gpt/
```

i get this error

```
(env) [wwilson@b-user-wwilson-m ~/persistent-disk/notebooks/pytorch-pretrained-BERT/examples]$ python run_gpt2.py --model_name_or_path=../roc_gpt/
Namespace(batch_size=-1, length=-1, model_name_or_path='../roc_gpt/', nsamples=1, seed=0, temperature=1.0, top_k=0, unconditional=False)
05/07/2019 09:58:24 - INFO - pytorch_pretrained_bert.tokenization_gpt2 -   loading special tokens file ../roc_gpt/special_tokens.txt
05/07/2019 09:58:24 - INFO - pytorch_pretrained_bert.tokenization_gpt2 -   loading vocabulary file ../roc_gpt/vocab.json
05/07/2019 09:58:24 - INFO - pytorch_pretrained_bert.tokenization_gpt2 -   loading merges file ../roc_gpt/merges.txt
05/07/2019 09:58:24 - INFO - pytorch_pretrained_bert.tokenization_gpt2 -   Special tokens {'_start_': 40478, '_delimiter_': 40479, '_classify_': 40480}
05/07/2019 09:58:24 - INFO - pytorch_pretrained_bert.modeling_gpt2 -   loading weights file ../roc_gpt/pytorch_model.bin
05/07/2019 09:58:24 - INFO - pytorch_pretrained_bert.modeling_gpt2 -   loading configuration file ../roc_gpt/config.json
05/07/2019 09:58:24 - INFO - pytorch_pretrained_bert.modeling_gpt2 -   Model config {
  "afn": "gelu",
  "attn_pdrop": 0.1,
  "embd_pdrop": 0.1,
  "initializer_range": 0.02,
  "layer_norm_epsilon": 1e-05,
  "n_ctx": 512,
  "n_embd": 768,
  "n_head": 12,
  "n_layer": 12,
  "n_positions": 512,
  "n_special": 3,
  "resid_pdrop": 0.1,
  "vocab_size": 40478
}

05/07/2019 09:58:26 - INFO - pytorch_pretrained_bert.modeling_gpt2 -   Weights of GPT2LMHeadModel not initialized from pretrained model: ['transformer.wte.weight', 'transformer.wpe.weight', 'transformer.ln_f.weight', 'transformer.ln_f.bias']
05/07/2019 09:58:26 - INFO - pytorch_pretrained_bert.modeling_gpt2 -   Weights from pretrained model not used in GPT2LMHeadModel: ['multiple_choice_head.linear.weight', 'multiple_choice_head.linear.bias', 'transformer.tokens_embed.weight', 'transformer.positions_embed.weight']
Traceback (most recent call last):
  File "run_gpt2.py", line 129, in <module>
    run_model()
  File "run_gpt2.py", line 77, in run_model
    model = GPT2LMHeadModel.from_pretrained(args.model_name_or_path)
  File "/mnt/notebooks/notebooks/pytorch-pretrained-BERT/env/lib/python3.6/site-packages/pytorch_pretrained_bert/modeling_gpt2.py", line 475, in from_pretrained
    "Error(s) in loading state_dict for {}:\n\t{}".format(model.__class__.__name__, "\n\t".join(error_msgs))
RuntimeError: Error(s) in loading state_dict for GPT2LMHeadModel:
        size mismatch for lm_head.decoder.weight: copying a param with shape torch.Size([40481, 768]) from checkpoint, the shape in current model is torch.Size([40478, 768]).
```

I'm guessing the first script is for gpt and the second one is for gpt2?
should i adjust the 2nd script to load the same classes as the trainer one to use the fine tuned gpt2 model?

thanks

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


```Traceback (most recent call last):
  File "question-generation/interact.py", line 238, in <module>
    run()
  File "question-generation/interact.py", line 144, in run
    model = GPT2LMHeadModel.from_pretrained(args.model_checkpoint)
  File "/usr/local/lib/python3.6/dist-packages/pytorch_pretrained_bert/modeling_gpt2.py", line 475, in from_pretrained
    "Error(s) in loading state_dict for {}:\n\t{}".format(model.__class__.__name__, "\n\t".join(error_msgs))
RuntimeError: Error(s) in loading state_dict for GPT2LMHeadModel:
	size mismatch for transformer.wte.weight: copying a param with shape torch.Size([50265, 768]) from checkpoint, the shape in current model is torch.Size([50257, 768]).
	size mismatch for lm_head.decoder.weight: copying a param with shape torch.Size([50265, 768]) from checkpoint, the shape in current model is torch.Size([50257, 768]).
```
I am having a similar issue? Any idea how to solve this? @Wingie 

+1

+1

+1


> ```
>   File "question-generation/interact.py", line 238, in <module>
>     run()
>   File "question-generation/interact.py", line 144, in run
>     model = GPT2LMHeadModel.from_pretrained(args.model_checkpoint)
>   File "/usr/local/lib/python3.6/dist-packages/pytorch_pretrained_bert/modeling_gpt2.py", line 475, in from_pretrained
>     "Error(s) in loading state_dict for {}:\n\t{}".format(model.__class__.__name__, "\n\t".join(error_msgs))
> RuntimeError: Error(s) in loading state_dict for GPT2LMHeadModel:
> 	size mismatch for transformer.wte.weight: copying a param with shape torch.Size([50265, 768]) from checkpoint, the shape in current model is torch.Size([50257, 768]).
> 	size mismatch for lm_head.decoder.weight: copying a param with shape torch.Size([50265, 768]) from checkpoint, the shape in current model is torch.Size([50257, 768]).
> ```
> 
> I am having a similar issue? Any idea how to solve this? @Wingie

Were you able to fix this? I am having this issue, with the same sizes as well.

same here!