Trying to train a GPT2 from scratch
Author: CNelias
Pull request: no

Hi ! 

I am trying to use a GPT2 architecture for musical applications and consequently need to train it from scratch. After a bit of googling I found that the issue #1714 already had "solved" the question but when I try the to run 
```Python
from transformers import GPT2Config, GPT2Model

NUMLAYER = 4
NUMHEAD = 4
SIZEREDUCTION = 10 #the factor by which we reduce the size of the velocity argument.
VELSIZE = int(np.floor(127/SIZEREDUCTION)) + 1 
SEQLEN=40 #size of data sequences.
EMBEDSIZE = 5 

config = GPT2Config(vocab_size = VELSIZE, n_positions = SEQLEN, n_embd = EMBEDSIZE, n_layer = NUMLAYER, n_ctx = SEQLEN, n_head = NUMHEAD)  
model = GPT2Model(config)
 ```
I get the following error : 
```Python
Traceback (most recent call last):

  File "<ipython-input-7-b043a7a2425f>", line 1, in <module>
    runfile('C:/Users/cnelias/Desktop/PHD/Swing project/code/script/GPT2.py', wdir='C:/Users/cnelias/Desktop/PHD/Swing project/code/script')

  File "C:\Users\cnelias\Anaconda3\lib\site-packages\spyder_kernels\customize\spydercustomize.py", line 786, in runfile
    execfile(filename, namespace)

  File "C:\Users\cnelias\Anaconda3\lib\site-packages\spyder_kernels\customize\spydercustomize.py", line 110, in execfile
    exec(compile(f.read(), filename, 'exec'), namespace)

  File "C:/Users/cnelias/Desktop/PHD/Swing project/code/script/GPT2.py", line 191, in <module>
    model = GPT2Model(config)

  File "C:\Users\cnelias\Anaconda3\lib\site-packages\transformers\modeling_gpt2.py", line 355, in __init__
    self.h = nn.ModuleList([Block(config.n_ctx, config, scale=True) for _ in range(config.n_layer)])

  File "C:\Users\cnelias\Anaconda3\lib\site-packages\transformers\modeling_gpt2.py", line 355, in <listcomp>
    self.h = nn.ModuleList([Block(config.n_ctx, config, scale=True) for _ in range(config.n_layer)])

  File "C:\Users\cnelias\Anaconda3\lib\site-packages\transformers\modeling_gpt2.py", line 223, in __init__
    self.attn = Attention(nx, n_ctx, config, scale)

  File "C:\Users\cnelias\Anaconda3\lib\site-packages\transformers\modeling_gpt2.py", line 109, in __init__
    assert n_state % config.n_head == 0
```
What does it mean and how can I solve it ?

Also more generally, is there a documentation on how to do a forward call with the GPT2 ? Can I define my own ```train()``` function or do I have to use the model's build-in function ? Am I forced to use a ```Dataset``` to do the training or can I feed it individual tensors ? 
I looked for it but couldn't find answer to these on the doc, but maybe I missed something

EDIT : Yes, I have already read the blogpost on ```huggingface.co``` but it omits too much informations and details to be usefull for my application :( 

Comments:
This [blogpost](https://huggingface.co/blog/how-to-train) might be interesting, have you seen it?

Yes, saddly the part in which I am interested in, namely, instantiating and training/testing from scratch with my own data are not almost not or not at all described.

Do you know if it is possible to feed individual tensors to the model ? And if so, how should the dimensions (batch, sequence etc..) be ordered ? I would like to write my own training function for more flexibility.

To answer my own question, everything can be found in the code, by reading the docstrings : https://github.com/huggingface/transformers/blob/v2.5.1/src/transformers/modeling_gpt2.py#L99


@johncwok Didu succeed in training the GPT2 model on your own dataset from scratch? 

I did, but it didn't produce very good results. Either my data is not good enough or I need more layers, but I have reach the max. of what my computer capacity can allow me 

Hi @johncwok, I plan to train gpt2 on my data. Do you mind to share your training script, along with the raw data and the code to preprocess it?