Error Loading Gpt-2 model after training from scratch.
Author: parthplc
Pull request: no

```
 Error(s) in loading state_dict for GPT2LMHeadModel:
	size mismatch for transformer.h.0.mlp.c_fc.weight: copying a param with shape torch.Size([768, 6]) from checkpoint, the shape in current model is torch.Size([768, 3072]).
```
I trained a GPT-2 model from scratch. When I tried loading the model using 
```
config = GPT2Config.from_json_file('../input/hindigpt/config.json')
# config.type_vocab_size=3072
model = GPT2LMHeadModel(config)
model.load_state_dict(torch.load('../input/hindigpt/pytorch_model.bin'))
```
I am getting the above error. Here is the copy of my config.json file
```
{
  "activation_function": "gelu_new",
  "architectures": [
    "GPT2LMHeadModel"
  ],
  "attn_pdrop": 0.1,
  "bos_token_id": 50256,
  "embd_pdrop": 0.1,
  "eos_token_id": 50256,
  "gradient_checkpointing": false,
  "initializer_range": 0.02,
  "layer_norm_epsilon": 1e-05,
  "model_type": "gpt2",
  "n_ctx": 1024,
  "n_embd": 768,
  "n_head": 12,
  "n_inner": 6,
  "n_layer": 6,
  "n_positions": 512,
  "resid_pdrop": 0.1,
  "summary_activation": null,
  "summary_first_dropout": 0.1,
  "summary_proj_to_labels": true,
  "summary_type": "cls_index",
  "summary_use_proj": true,
  "vocab_size": 36021
}
```
Can anyone help me with this?


Comments:
Hello! How did you train your GPT-2 from scratch? Which framework did you use?

> Hello! How did you train your GPT-2 from scratch? Which framework did you use?

Used PyTorch framework.

Used this script
```
import os
import gc
import glob
import torch
import pickle
import joblib
from tqdm.auto import tqdm
from pathlib import Path
from tokenizers import ByteLevelBPETokenizer
from transformers import GPT2Tokenizer
import torch
from transformers import GPT2TokenizerFast
from transformers import GPT2LMHeadModel
from transformers import DataCollatorForLanguageModeling
from transformers import TextDataset
from transformers import GPT2Config
from transformers import Trainer, TrainingArguments


tokenizer = GPT2Tokenizer.from_pretrained('hindi/')
vocab_size = tokenizer.vocab_size 
print(vocab_size)
print(torch.cuda.is_available())


config = GPT2Config(
    vocab_size=vocab_size  
)

tokenizer = GPT2TokenizerFast.from_pretrained("hindi/", max_len=512)
model = GPT2LMHeadModel(config=config)
print(model.num_parameters())

print("Now let's build our training Dataset")

dataset = TextDataset(
    tokenizer=tokenizer,
    file_path="data/train.txt",
    block_size=132,
)

print("Start Training")

data_collator = DataCollatorForLanguageModeling(
    tokenizer=tokenizer, mlm=True, mlm_probability=0.15
)

print("Trainer Classes")

training_args = TrainingArguments(
    output_dir="hindi/",
    overwrite_output_dir=True,
    num_train_epochs=1,
    per_gpu_train_batch_size=64,
    save_steps=10_000,
    save_total_limit=2,
)

trainer = Trainer(
    model=model,
    args=training_args,
    data_collator=data_collator,
    train_dataset=dataset,
    prediction_loss_only=True,
)

trainer.train()

trainer.save_model("hindi/")
print("done")
```
for training @LysandreJik